Re: IEP-14: Ignite failures handling (Discussion)

[Nick Pordash]

I can tell you as a user that if any library I was using in my application
called System.exit without my consent would result in a lot of frustration.

If ignite enters an unrecoverable state then I think that is something that
should be observable locally, similar to node segmentation and then the
application can decide the best course of action.

Of course, if ignite was started as a standalone process do what you think
is best, but don't think you can kill the process without backlash from
users if it's running in embedded mode.

- Nick

On Tue, Mar 13, 2018, 5:12 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Ivan,
>
> If grid hangs, graceful shutdown would most likely hang as well. Almost
> never you can recover from a bad state using graceful procedures.
>
> I agree that we should not create two defaults, especially in this case.
> It's not even strictly defined what is embedded node in Ignite. For
> example, if I start it using a custom main class and/or custom script
> instead of ignite.sh, would it be embedded or standalone node?
>
> -Val
>
> On Tue, Mar 13, 2018 at 4:58 PM, Ivan Rakov  wrote:
>
> > One more note: "kill if standalone, stop if embedded" differs from what
> > you are suggesting "try graceful, then kill process regardless" only in
> > case when graceful shutdown hangs.
> > Do we have understanding, how often does graceful shutdown hang?
> > Obviously, *grid hang* is often case, but it shouldn't be messed with
> > *graceful shutdown hang*. From my experience, if something went wrong,
> > users just prefer to do kill -9  because it's much more reliable and
> easy.
> > Probably, in most of cases when kill -9 worked, graceful stop would have
> > worked as well - we just don't have such statistics.
> > It may be bad example, but: in our CI tests we intentionally break grid
> in
> > many harsh ways and perform a graceful stop after the test execution, and
> > it doesn't hang - otherwise we'd see many "Execution timeout" test suite
> > hangs.
> >
> > Best Regards,
> > Ivan Rakov
> >
> >
> > On 14.03.2018 2:24, Dmitriy Setrakyan wrote:
> >
> >> On Tue, Mar 13, 2018 at 7:13 PM, Ivan Rakov 
> >> wrote:
> >>
> >> I just would like to add my +1 for "kill if standalone, stop if
> embedded"
> >>> default option. My arguments:
> >>>
> >>> 1) Regarding "If Ignite hangs - it will likely be impossible to stop":
> >>> Unfortunately, it's true that Ignite can hang during stop procedure.
> >>> However, most of failures described under IEP-14 (storage IO
> exceptions,
> >>> death of critical system worker thread, etc) normally shouldn't turn
> node
> >>> into "impossible to stop" state. Turning into that state is a bug
> >>> itself. I
> >>> guess that we shouldn't choose system behavior on the basis of known
> >>> bugs.
> >>>
> >>
> >> The whole discussion is about protecting against force-major issues,
> >> including Ignite bugs. You are assuming that a user application will
> >> somehow continue to function if an Ignite node is stopped. In most cases
> >> it
> >> will just freeze itself and cause the rest of the application to hang.
> >>
> >> Again, "kill+stop" is the most deterministic and the safest default
> >> behavior. Try a graceful shutdown (which will make restart easier), and
> >> then kill the process regardless.
> >>
> >> Note that we are arguing about the default behavior. If a user does not
> >> like this default, then this user can change it to another behavior.
> >>
> >>
> >> 2) User might want to handle Ignite node crash before shutting down the
> >>> whole JVM - raise alert, close external resources, etc
> >>>
> >>> Very unlikely, but if a user is this advanced, then this user can
> change
> >> the default behavior. Most users will not even know how to configure
> such
> >> custom shutdown behavior and would prefer an automatic kill.
> >>
> >> 3) IEP-14 document has important notes: "More than one Ignite node could
> >> be
> >>
> >>> started in one JVM process" and "Different nodes in one JVM process
> could
> >>> belong to different clusters". This is possible only in embedded mode.
> I
> >>> think, we shouldn't shock user by sudden JVM halt (possibly, along with
> >>> another healthy nodes) if there's a chance of successful node stop.
> >>>
> >>> Has anyone actually seen a real example of that? I have not. This
> >> scenario
> >> is extremely unlikely and should not define the default behavior. Again,
> >> if
> >> a user is so advanced to come up with such a sophisticated deployment,
> >> then
> >> the same user should be able to set different default behaviors for
> >> different clusters.
> >>
> >>
> >
>

Re: Ignite node crashes after one query fetches many entries from cache

[Nick Pordash]

Ray,

I think to avoid excessive GC and OOM you could try switching to a lazy
result set:
https://apacheignite-sql.readme.io/docs/performance-and-debugging#result-set-lazy-load

- Nick

On Wed, Nov 29, 2017, 7:19 AM Anton Vinogradov 
wrote:

> Ray,
>
> Seems you're looking
> for org.apache.ignite.cache.query.SqlFieldsQuery#timeout?
>
> On Tue, Nov 28, 2017 at 5:30 PM, Alexey Kukushkin <
> kukushkinale...@gmail.com> wrote:
>
>> Ignite Developers,
>>
>> I know community is developing an "Internal Problems Detection" feature
>> .
>> Do you know if it addresses the problem Ray described below? May be we
>> already have a setting to prevent this from happening?
>>
>> On Tue, Nov 28, 2017 at 5:13 PM, Ray  wrote:
>>
>>> I try to fetch all the results of a table with billions of entries using
>>> sql
>>> like this "select * from table_name".
>>> As far as my understanding, Ignite will prepare all the data on the node
>>> running this query then return the results to the client.
>>> The problem is that after a while, the node crashes(probably because of
>>> long
>>> GC pause or running out of memory).
>>> Is node crashing the expected behavior?
>>> I mean it's unreasonable that Ignite node crashes after this kind of
>>> query.
>>>
>>> From my experience with other databases,  running this kind of full table
>>> scan will not crash the node.
>>>
>>> The optimal way for handling this kind of situation is Ignite node stays
>>> alive, the query will be stopped by Ignite node when the node find out it
>>> will run out of memory soon.
>>> Then an error response shall be returned to the client.
>>>
>>> Please advice me if this mechanism already exists and there is hidden
>>> switch
>>> to turn it on.
>>> Thanks
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
>>
>>
>> --
>> Best regards,
>> Alexey
>>
>
>

Re: Cluster auto activation design proposal

[Nick Pordash]

How is a user expected to produce a collection of ClusterNode prior to all
of the expected nodes joining? Users don't create instances of this, so as
far as I can tell it would have to be retrieved from IgniteCluster.
However, would doing that and calling the proposed method be really any
different than calling Ignite.activate and using the current set of server
nodes as that collection?

>From a user's perspective is it really necessary that specific nodes need
to be identified vs saying that they expect N server nodes to be in the
cluster for auto activation?

-Nick

On Wed, Aug 30, 2017, 1:23 AM Sergey Chugunov 
wrote:

> Dmitriy,
>
> Now I see your point and I think you're right.
>
> We can give end-user a simple setter like
> Ignite::activation::setInitialActivationNodes(Collection
> nodes) to provide collection of nodes that grid must reach to activate
> automatically.
>
> And then using the collection we'll create BaselineTopology abstraction
> internally.
>
> As a result user won't be exposed to our internal abstractions and will
> work with intuitive concept of collection of nodes.
>
> Thanks,
> Sergey.
>
>
> On Mon, Aug 28, 2017 at 4:39 PM, Dmitriy Setrakyan 
> wrote:
>
> > Sergey, the interface you are suggesting is internal, not external. Why
> > should user ever see it or care about it?
> >
> > D.
> >
> > On Mon, Aug 28, 2017 at 3:18 PM, Sergey Chugunov <
> > sergey.chugu...@gmail.com>
> > wrote:
> >
> > > Dmitriy,
> > >
> > > It was my misunderstanding, I believe that setter is not enough and we
> > need
> > > a full-fledged entity.
> > >
> > > We should also be able to check if BLTs are compatible. Interface looks
> > > like this and use case for this functionality is described below.
> > >
> > > interface BaselineTopology {
> > >Collection nodes();
> > >boolean isCompatibleWith(BaselineTopology blt);
> > > }
> > >
> > > Let's consider the following scenario:
> > >
> > >1. We have a grid with N nodes: it is up, active and has data in it.
> > ->
> > >BLT #1 created.
> > >2. We shutdown the grid. Then divide it into two parts: Part1_grid
> and
> > >Part2_grid.
> > >3. We start and activate Part1_grid . Topology has changed -> BLT#2
> > >created.
> > >After that we shutdown that Part1_grid.
> > >4. We start and activate Part2_grid. Topology also has changed ->
> > BLT#3
> > >created.
> > >5. Then we start Part1_grid and it's nodes try to join Part2_grid.
> > >
> > >
> > > If join is successful we have an undefined state of the resulting grid:
> > > values for the same key may (and will) differ between grid parts.
> > >
> > > So to prevent this we should keep nodes with BLT#2 from joining the
> grid
> > > with BLT#3. And we should fail nodes with an error message.
> > >
> > > Thanks,
> > > Sergey.
> > >
> > >
> > >
> > >
> > > On Wed, Aug 23, 2017 at 5:47 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > Sergey, I am still confused. What is the BaselineTopology interface
> in
> > > your
> > > > example? I thought that you agreed with me that we simply need a
> setter
> > > for
> > > > activation nodes, no?
> > > >
> > > > D.
> > > >
> > > > On Tue, Aug 22, 2017 at 4:47 AM, Sergey Chugunov <
> > > > sergey.chugu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > As I understand you use the term "minimalActivationNodes" as a
> > synonym
> > > > for
> > > > > BaselineTopology concept.
> > > > > In that case I agree with you that we can replace both "establish*"
> > > > methods
> > > > > with a simple setter method (see below in summary).
> > > > >
> > > > > Summing up the whole discussion I see the functionality as
> following:
> > > > >
> > > > > New concept BaselineTopology is introduced. The main features it
> > > enables
> > > > > are:
> > > > >
> > > > >1. automatic activation of cluster;
> > > > >
> > > > >2. easy management of cluster topology changes (planned nodes
> > > > >maintenance, adding new nodes etc);
> > > > >
> > > > >3. eliminating of rebalancing traffic on short-term node
> failures.
> > > > >
> > > > >
> > > > > Use cases to create BLT:
> > > > >
> > > > >1. user starts up new cluster of desired number of nodes and
> > > activates
> > > > >it using existing API. BLT is created with all nodes presented
> in
> > > the
> > > > >cluster at the moment of activation, no API is needed;
> > > > >
> > > > >2. user prepares BLT using web-console or visor CMD tools and
> sets
> > > it
> > > > to
> > > > >the cluster. New API setter is needed:
> > > > >Ignite.activation().setBaselineTopology(BaselineTopology blt);
> > > > >
> > > > >3. user provides via static configuration a list of nodes that
> are
> > > > >expected to be in the cluster.
> > > > >User starts nodes one by one; when all preconfigured nodes are
> > > started
> > > > >cluster is activated and BLT is created.
> > > > >

Re: Policy for update third-party dependencies

[Nick Pordash]

Hi Val,

Pretty much, with obvious exceptions being integration modules with other
projects. If the dependency is well isolated, then shading could be
beneficial.

I've also had to do this for client libraries operating inside other
frameworks (I've had to shade netty to avoid conflicting with user code).
It's a good alternative since relying on things like OSGi isn't all that
practical due to lack of widespread adoption.

-Nick

On Mon, Aug 21, 2017, 10:48 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Nick,
>
> Do you suggest to build and deploy uber-jars that has no external
> dependencies?
>
> -Val
>
> On Sun, Aug 20, 2017 at 1:02 PM, Nick Pordash <nickpord...@gmail.com>
> wrote:
>
> > If the dependency is not exposed by the public API then another
> alternative
> > is to simply shade the artifact and then this becomes a non-issue for
> > users.
> >
> > Considering Ignite is a platform that executes user code via compute and
> > service grid I personally think it would be good to minimize the number
> of
> > dependencies that can potentially conflict with user code.
> >
> > -Nick
> >
> > On Sun, Aug 20, 2017, 11:51 AM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Guys,
> > >
> > > Keep in mind that some projects can use *older* version of third-party
> > > libraries as well, and dependency upgrade can break them. In other
> words,
> > > dependency upgrade is in many cases an incompatible change for us, so
> we
> > > should do this with care.
> > >
> > > Unless there is a specific reason to upgrade a specific dependency, I
> > think
> > > it's better to postpone it until major version.
> > >
> > > -Val
> > >
> > > On Sun, Aug 20, 2017 at 5:04 AM 李玉珏@163 <18624049...@163.com> wrote:
> > >
> > > > If the third party library is incompatible with the new version and
> the
> > > > old version (such as lucene3.5.0-5.5.2), and the dependent version of
> > > > Ignite is older, it may cause conflicts in the user's system.
> > > > For such scenarios, I think that updating third-party dependencies's
> > > > major version is valuable.
> > > >
> > > >
> > > > 在 2017/8/17 上午8:26, Denis Magda 写道:
> > > > > I would respond why do we need to update? Some bug, new
> capabilities,
> > > > security breach? Alexey K., please shed some light on this.
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > >> On Aug 16, 2017, at 5:12 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >
> > > > wrote:
> > > > >>
> > > > >> On Wed, Aug 16, 2017 at 5:02 PM, Denis Magda <dma...@apache.org>
> > > wrote:
> > > > >>
> > > > >>> Honestly, I wouldn’t touch a dependency if it works like a charm
> > and
> > > > >>> nobody requested us to migrate to a new version.
> > > > >>>
> > > > >>> Why do you need to update Apache Common coded?
> > > > >>>
> > > > >> Not sure I agree. Why not update it?
> > > > >>
> > > > >>
> > > > >>>
> > > > >>> —
> > > > >>> Denis
> > > > >>>
> > > > >>>> On Aug 16, 2017, at 10:36 AM, Alexey Kuznetsov <
> > > akuznet...@apache.org
> > > > >
> > > > >>> wrote:
> > > > >>>> Done
> > > > >>>>
> > > > >>>> https://issues.apache.org/jira/browse/IGNITE-6090
> > > > >>>>
> > > > >>>> On Wed, Aug 16, 2017 at 8:01 PM, Dmitriy Setrakyan <
> > > > >>> dsetrak...@apache.org>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> The answer is Yes, we should update. Jira ticket assigned to
> the
> > > next
> > > > >>>>> release should be enough in my view.
> > > > >>>>>
> > > > >>>>> D.
> > > > >>>>>
> > > > >>>>> On Wed, Aug 16, 2017 at 2:38 AM, Alexey Kuznetsov <
> > > > >>> akuznet...@apache.org>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Hi, All!
> > > > >>>>>>
> > > > >>>>>> Do we have any policy for updating third-party dependencies?
> > > > >>>>>>
> > > > >>>>>> For example, I found that we are using very old  Apache Common
> > > codec
> > > > >>>>> v.1.6
> > > > >>>>>> (released in 2011)
> > > > >>>>>> And latest is Apache Common codec v.1.10
> > > > >>>>>>
> > > > >>>>>> Do we need to update to new versions from time to time?
> > > > >>>>>> And how?
> > > > >>>>>>
> > > > >>>>>> Just create JIRA issue, update pom.xml and run all tests on
> TC -
> > > > will
> > > > >>> be
> > > > >>>>>> enough?
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>> Alexey Kuznetsov
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> Alexey Kuznetsov
> > > > >>>
> > > >
> > > >
> > > >
> > >
> >
>

Re: Ignite: configuration changes at runtime

[Nick Pordash]

The notion of saving changes to a configuration file does not make sense
for lots of use cases, Kubernetes is one example. Anything in the
container's filesystem is considered transient since it will get destroyed
when a pod is restarted.

I would think that runtime changes should be managed in the cluster
metadata somewhere instead of expecting them to be exported elsewhere like
a file. If new nodes join the cluster then runtime changes received as a
part of a node join operation would get applied prior to node activation
from completing. These would override any node configuration provided on
node startup, regardless of how it was configured.

IMO if someone wants to make runtime changes AND propagate those changes to
wherever they are doing external configuration management then perhaps that
should be left as an exercise for the user. There are so many ways that
this could be done and often what is sitting on the filesystem would easily
get overridden by some other mechanism (f.e ansible, salt, puppet, chef,
container orchestration systems, etc).

-Nick

On Tue, Aug 22, 2017, 6:54 PM Dmitriy Setrakyan 
wrote:

> On Tue, Aug 22, 2017 at 4:43 AM, Yakov Zhdanov 
> wrote:
>
> > >Not really. What if the user configured Ignite from Java API, without
> any
> > >XML. What will be saved? To prevent this ambiguity, I think the cleanest
> > >way is to ask user to save the configuration to a file explicitly.
> >
> > I think it does not matter too much how the grid was configured. In any
> > case IgniteConfiguration object is created. And we will be allowing to
> > change only very few parameters that are mostly of primitive type or
> enum.
> >
> > Once changed configuration is saved to metastore. When restarting we
> should
> > apply it by default or allow user to override it with provided one.
> >
>
> I would prefer that users explicitly specify which configuration file to
> use on startup. This will prevent any magic. Still think that explicit call
> to saveConfigurationFile(...) is the best approach.
>

Re: Policy for update third-party dependencies

[Nick Pordash]

If the dependency is not exposed by the public API then another alternative
is to simply shade the artifact and then this becomes a non-issue for
users.

Considering Ignite is a platform that executes user code via compute and
service grid I personally think it would be good to minimize the number of
dependencies that can potentially conflict with user code.

-Nick

On Sun, Aug 20, 2017, 11:51 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Guys,
>
> Keep in mind that some projects can use *older* version of third-party
> libraries as well, and dependency upgrade can break them. In other words,
> dependency upgrade is in many cases an incompatible change for us, so we
> should do this with care.
>
> Unless there is a specific reason to upgrade a specific dependency, I think
> it's better to postpone it until major version.
>
> -Val
>
> On Sun, Aug 20, 2017 at 5:04 AM 李玉珏@163 <18624049...@163.com> wrote:
>
> > If the third party library is incompatible with the new version and the
> > old version (such as lucene3.5.0-5.5.2), and the dependent version of
> > Ignite is older, it may cause conflicts in the user's system.
> > For such scenarios, I think that updating third-party dependencies's
> > major version is valuable.
> >
> >
> > 在 2017/8/17 上午8:26, Denis Magda 写道:
> > > I would respond why do we need to update? Some bug, new capabilities,
> > security breach? Alexey K., please shed some light on this.
> > >
> > > —
> > > Denis
> > >
> > >> On Aug 16, 2017, at 5:12 PM, Dmitriy Setrakyan  >
> > wrote:
> > >>
> > >> On Wed, Aug 16, 2017 at 5:02 PM, Denis Magda 
> wrote:
> > >>
> > >>> Honestly, I wouldn’t touch a dependency if it works like a charm and
> > >>> nobody requested us to migrate to a new version.
> > >>>
> > >>> Why do you need to update Apache Common coded?
> > >>>
> > >> Not sure I agree. Why not update it?
> > >>
> > >>
> > >>>
> > >>> —
> > >>> Denis
> > >>>
> >  On Aug 16, 2017, at 10:36 AM, Alexey Kuznetsov <
> akuznet...@apache.org
> > >
> > >>> wrote:
> >  Done
> > 
> >  https://issues.apache.org/jira/browse/IGNITE-6090
> > 
> >  On Wed, Aug 16, 2017 at 8:01 PM, Dmitriy Setrakyan <
> > >>> dsetrak...@apache.org>
> >  wrote:
> > 
> > > The answer is Yes, we should update. Jira ticket assigned to the
> next
> > > release should be enough in my view.
> > >
> > > D.
> > >
> > > On Wed, Aug 16, 2017 at 2:38 AM, Alexey Kuznetsov <
> > >>> akuznet...@apache.org>
> > > wrote:
> > >
> > >> Hi, All!
> > >>
> > >> Do we have any policy for updating third-party dependencies?
> > >>
> > >> For example, I found that we are using very old  Apache Common
> codec
> > > v.1.6
> > >> (released in 2011)
> > >> And latest is Apache Common codec v.1.10
> > >>
> > >> Do we need to update to new versions from time to time?
> > >> And how?
> > >>
> > >> Just create JIRA issue, update pom.xml and run all tests on TC -
> > will
> > >>> be
> > >> enough?
> > >>
> > >> --
> > >> Alexey Kuznetsov
> > >>
> > 
> > 
> >  --
> >  Alexey Kuznetsov
> > >>>
> >
> >
> >
>

Re: IGNITE-5123 Review

[Nick Pordash]

Hi Dmitriy,

The ticket was a regression from 1.9 to 2.0. I don't think anyone would be
expecting the behavior in 2.0 as it doesn't align with the javadoc and has
only been broken since the 2.0 release.

-Nick

On Wed, Jul 19, 2017, 6:55 AM Dmitry Pavlov  wrote:

> Hi Evgeniy,
>
> Thank you. Ignite Basic is one suite from approximately 80 suites that
> covers Ignite by automated tests. Which is why I suggested to use RunAll
> chain in ignite 2.0 group. Yes, several tests may fail, especially if it is
> flaky tests or failure is related to the specific JIRA issue.
>
> About change itself: This change seems to be very impact. There is
> possiblity that many of existing plugins relies on existing order of
> initialization. This change may break plugin initialization in unexpected
> manner.
>
> Could we
> - fix javadoc according to existing order in code
> - find out new solution?
>
> Sincerely,
> Dmitriy Pavlov
>
>
> ср, 19 июл. 2017 г. в 16:40, Evgeniy Ignatiev  >:
>
> >
> >
> http://ci.ignite.apache.org/viewLog.html?buildId=720722=buildResultsDiv=IgniteTests_IgniteBasic
> > - this one - there seem to be no new failed platform tests, other failed
> > tests seem to fail in several other reviews too and are unrelated to my
> > changes.
> >
> >
> > On 19.07.2017 17:35, Dmitry Pavlov wrote:
> > > Hi Evgeniy,
> > >
> > > I was not able to find Teamcity run for this change.
> > > Could you please run http://ci.ignite.apache.org test for example on
> > branch
> > > pull/2285/head using 'Ignite 2.0 Tests' target 'Run All'.
> > > Or could you please share link to previous run on this changes?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > ср, 19 июл. 2017 г. в 15:26, Anton Vinogradov :
> > >
> > >> Igniters,
> > >>
> > >> Could somebody review the fix today?
> > >>
> > >> On Wed, Jul 19, 2017 at 1:30 PM, Evgeniy Ignatiev <
> > >> yevgeniy.ignat...@gmail.com> wrote:
> > >>
> > >>> Hello, Igniters.
> > >>>
> > >>> Could anyone review my request - https://issues.apache.org/jira
> > >>> /browse/IGNITE-5123? - My previous pings seems to got lost.
> > >>>
> > >>> Best regards,
> > >>>
> > >>> Yevgeniy
> > >>>
> > >>>
> >
> >
>

[jira] [Created] (IGNITE-5123) Ignite.cache(String) returns null in PluginProvider.onIgniteStart()

[Nick Pordash (JIRA)]

Nick Pordash created IGNITE-5123:


 Summary: Ignite.cache(String) returns null in 
PluginProvider.onIgniteStart()
 Key: IGNITE-5123
 URL: https://issues.apache.org/jira/browse/IGNITE-5123
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.0
Reporter: Nick Pordash


Given an Ignite node that has pre-configured caches (via 
IgniteConfiguration.setCacheConfiguration) if you try to obtain a reference to 
the cache instance in PluginProvider.onIgniteStart() you'll get a null 
reference.

@Override
public void onIgniteStart() throws IgniteCheckedException {
ignite.cacheNames().forEach(name -> {
assert ignite.cache(name) != null : "Cache is null: " + name;
});
}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)