Re: Add IgniteInvalidGridStateException instead of IllegalStateException in KernalGateway

[Guru Stron]

Great find, Pavel!

Thank you, will go with it than.

On Mon, 31 Aug 2020 at 22:45, Pavel Tupitsyn  wrote:

> Actually, we already have IgniteIllegalStateException,
> it exists exactly for situations like this.
>
> Let's just change GridKernalGatewayImpl to throw
> IgniteIllegalStateException
> instead of IllegalStateException and handle that in .NET
>
> On Sat, Aug 1, 2020 at 10:32 AM Pavel Tupitsyn 
> wrote:
>
> > Sounds good to me.
> > We can include grid state as a property of that exception too.
> >
> > On Fri, Jul 31, 2020 at 5:32 PM Guru Stron 
> > wrote:
> >
> >> Dear Igniters,
> >>
> >> I'm working on a bug in .NET code that handles transaction states (jira
> >> ticket ). And I
> need
> >> a
> >> way to handle stopped grid exception, so I think it will make sense to
> >> throw a specialized exception type in this case instead of a generic
> >> IllegalStateException
> >> <
> >>
> https://github.com/apache/ignite/blob/d486f24b1be3f08e8b1e5e9992c4c8718e169299/modules/core/src/main/java/org/apache/ignite/internal/GridKernalGatewayImpl.java#L95
> >> >
> >> .
> >>
> >> Any objections?
> >>
> >> Best regards!
> >>
> >
>

Command line interface to manage distributed properties

[Taras Ledkov]

Hi,

Motivation: we have to manage SQL distributed property by command line 
and introduce common approach to manage distributed properties.

Issue: IGNITE-13186 (see [1])

My proposal is:

Property classes & DistributedConfigurationProcessor changes (see PR [2]):
- introduce PublicProperty interface and implements it at the 
PublicSimpleProperty;
- SimpleDistributedPublicProperty. If we create the instance of this 
class the property can be managed by command line.


Command line interface:
--property list - prints list of the available properties with 
description, e.g.:

    sql.disabledFunctions : Disabled SQL functions
    sql.defaultQueryTimeout : Default query timeout
--property get --name  - prints the property value
--property set --name  --val  - change the 
property value.


Possible we have to add the command:
--property reset --name  - reset property to default value.

Please your comments.
Please pay your attention to concept & design of the publishing a 
property by 'PublicProperty' and set of the new commands.


[1]. https://issues.apache.org/jira/browse/IGNITE-13186
[2]. https://github.com/apache/ignite/pull/8208

--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: Proposal of new event QUERY_EXECUTION_EVENT

[Dmitrii Ryabov]

I agree with Max, we need to add a separate event for starting query
execution, and EVT_CACHE_QUERY_EXECUTED shouldn't be deprecated,
because it is another case - it is fired when cache query was
successfully finished.

> Would the event notification be synchronous? In which thread?
As Max said, synchronicity depends on implementation. As I see, we
don't use a separate thread for any record calls.

> What happens in case the event listener fails?
Exceptions are logged by `U.error(...)` call. Errors are thrown out.

> Should we discuss this within this topic?
I suggest to separate adding a new event and configuring existing events.

пн, 20 июл. 2020 г. в 14:37, Max Timonin :
>
> Looks like EVT_CACHE_QUERY_EXECUTED just works for different use cases:
> 1. it relates to a specific cache (Event for SQL queries looks different as
> it could contain multiple caches or none of them);
> 2. Also the EVT_CACHE_QUERY_EXECUTED event fires multiple times for
> distributed queries, see GridMapQueryExecutor class (For SQL query it's
> required to fire once independently on how many nodes are affected).
>
> So there are different patterns for events. I think
> EVT_CACHE_QUERY_EXECUTED should not be deprecated or changed.
>
> > What happens in case the event listener fails?
> > Would the event notification be synchronous?
> It depends on how other events are implemented. As I see for the
> EVT_CACHE_QUERY_EXECUTED event - it's synchronous, and listener errors
> aren't handled.
>
> I think these questions are related to GridEventStorageManager as it just
> provides an API for recording events. Internal implementations (sync
> / async, error handling) is not related to an event, is it?
>
> > I have some doubts about provide text of a query even with
> hidden arguments, probably it should be configured due to it could lead
> to security leak
> Currently event EVT_CACHE_QUERY_EXECUTED provides a sql clause without
> limitations. If we're going to provide some restrictions it will require
> additional investigation. I see at least 2 configurations here:
> 1. Ignite can be configured to hide clase, params only or nothing for all
> listeners;
> 2. Only authorized listeners can subscribe to the event.
>
> Should we discuss this within this topic?
>
> On Mon, Jul 20, 2020 at 1:55 PM Юрий  wrote:
>
> > In my opinion existing events EVT_CACHE_QUERY_EXECUTION_STARTED should be
> > deprecated and added two new for start and for end of queries which should
> > cover all SQL query types.
> > I have some doubts about provide text of a query even with hidden
> > arguments, probably it should be configured due to it could lead to
> > security leak
> >
> > пн, 20 июл. 2020 г. в 12:49, Stanislav Lukyanov :
> >
> > > Maksim,
> > >
> > > Can we change the EVT_CACHE_QUERY_EXECUTED to fire earlier? Or should
> > > there be an EVT_CACHE_QUERY_EXECUTION_STARTED for the query start, while
> > > the old event would continue to work for query finish?
> > > I think the new event needs to either reuse the old one, or be its mirror
> > > for the query start. It also means that we probably should resolve the
> > > issues you've listed.
> > >
> > > Would the event notification be synchronous? In which thread?
> > Asynchronous
> > > is generally preferred - would it work?
> > >
> > > What happens in case the event listener fails?
> > >
> > > Thanks,
> > > Stan
> > >
> > > > On 16 Jul 2020, at 18:49, Denis Magda  wrote:
> > > >
> > > > Taras, Yury, Ivan,
> > > >
> > > > Could you please join this thread and share your thoughts? Do we
> > already
> > > > have any plans on tracking of the DDL and DML queries?
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Wed, Jul 15, 2020 at 12:09 AM Max Timonin 
> > > > wrote:
> > > >
> > > >> Hi Denis, thanks for the answer!
> > > >>
> > > >> We already checked EVT_CACHE_QUERY_EXECUTED and found that it works
> > > only in
> > > >> cases:
> > > >> 1. Scan queries and Select queries (common pattern is access to cache
> > > >> data);
> > > >> 2. This event triggers only if query execution succeeds, in case of
> > > failure
> > > >> while execution this event won't fire.
> > > >>
> > > >> Our additional requirements are to protocol queries:
> > > >> 1. that aren't cache related (for example, alter user);
> > > >> 2. that relate to multiple caches (while EVT_CACHE_QUERY_EXECUTED have
> > > >> field cacheName related to specific cache);
> > > >> 3. we need to protocol also DDL and DML queries.
> > > >>
> > > >> Regards,
> > > >> Maksim
> > > >>
> > > >> On Tue, Jul 14, 2020 at 10:20 PM Denis Magda 
> > wrote:
> > > >>
> > > >>> Hi Max,
> > > >>>
> > > >>> Could you check if the EVT_CACHE_QUERY_EXECUTED event is what you're
> > > >>> looking for?
> > > >>>
> > > >>>
> > > >>
> > >
> > https://www.gridgain.com/docs/latest/developers-guide/events/events#cache-query-events
> > > >>>
> > > >>> -
> > > >>> Denis
> > > >>>
> > > >>>
> > > >>> On Fri, Jul 10, 2020 at 3:54 AM Max Timonin  > >
> > > >>> wrote:
> > > >>>
> > > 

[jira] [Created] (IGNITE-13396) ignite.queue return null instead of throwing IgniteException as declared in the javadoc

[Vladimir Pligin (Jira)]

Vladimir Pligin created IGNITE-13396:


 Summary: ignite.queue return null instead of throwing 
IgniteException as declared in the javadoc
 Key: IGNITE-13396
 URL: https://issues.apache.org/jira/browse/IGNITE-13396
 Project: Ignite
  Issue Type: Bug
  Components: data structures
Affects Versions: 2.8.1
Reporter: Vladimir Pligin
 Fix For: 2.10


{code:java}
Ignite ignite = Ignition.start();
IgniteQueue queue = ignite.queue("Queue", 0, null);
ignite.close();
{code}
This code returns null. We need to fix it or align the javadoc.

Here's the initial discussion: 
[http://apache-ignite-users.70518.x6.nabble.com/Ignite-Queue-Documentation-or-Code-defect-td33703.html]
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Incorrect TeamCity OpenCensus suite configuration

[Mikhail Petrov]

Nikolay, thanks a lot. Now everything works just fine.

On 02.09.2020 14:08, Nikolay Izhikov wrote:

Hello, Mikhail.

Thanks.
Done.

https://ci.ignite.apache.org/viewLog.html?buildId=5579056=queuedBuildOverviewTab


2 сент. 2020 г., в 11:41, Mikhail Petrov  написал(а):

Igniters,

Not all tests from the IgniteOpenCensusSuite are currently running on TeamCity. 
It seems that it due to incorrect TC settings - TEST_SUITE value for OpenCensus 
suite configuration should be IgniteOpenCensusSuite but the current value is 
OpenCensusMetricExporterSpiTest [1].

Unless this is done on purpose, can anyone help me to fix this issue?

[1] - 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_OpenCensus=buildTypeSettings_IgniteTests24Java8=%3Cdefault%3E

Re: Incorrect TeamCity OpenCensus suite configuration

[Nikolay Izhikov]

Hello, Mikhail.

Thanks.
Done.

https://ci.ignite.apache.org/viewLog.html?buildId=5579056=queuedBuildOverviewTab

> 2 сент. 2020 г., в 11:41, Mikhail Petrov  написал(а):
> 
> Igniters,
> 
> Not all tests from the IgniteOpenCensusSuite are currently running on 
> TeamCity. It seems that it due to incorrect TC settings - TEST_SUITE value 
> for OpenCensus suite configuration should be IgniteOpenCensusSuite but the 
> current value is OpenCensusMetricExporterSpiTest [1].
> 
> Unless this is done on purpose, can anyone help me to fix this issue?
> 
> [1] - 
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_OpenCensus=buildTypeSettings_IgniteTests24Java8=%3Cdefault%3E
> 
> 
> 

[jira] [Created] (IGNITE-13395) Get rid of partition group reservations.

[Alexey Scherbakov (Jira)]

Alexey Scherbakov created IGNITE-13395:
--

 Summary: Get rid of partition group reservations.
 Key: IGNITE-13395
 URL: https://issues.apache.org/jira/browse/IGNITE-13395
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.9
Reporter: Alexey Scherbakov
 Fix For: 2.10


Currently we have two ways to prevent a partition from being evicted if it's no 
longer owner by affinity: 

1. "normal" reservation using GridDhtLocalPartition#reserve/release
2. "group" reservation using GridDhtPartitionsReservation.

The second is used only for SQL queries to ensure consistent result on topology 
change and adds a lot of complexity to SQL engine and partition eviction code. 
In fact, the same can be achieved using only first type of reservation.

I propose to get rid of group reservations using the following algorythm:

1. On PME all owning partitions are normally reserved.
2. A query counter for topology version is introduced.
3. If a query counter reaches zero for current topVer and where is a new topVer 
incompatible with current, previously reserved partitions are released.

Using such approach it's possible to totally remove any reservation management 
from SQL engine (except maintaining a counter), make it more simple and 
efficient.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Incorrect TeamCity OpenCensus suite configuration

[Mikhail Petrov]

Igniters,

Not all tests from the IgniteOpenCensusSuite are currently running on 
TeamCity. It seems that it due to incorrect TC settings - TEST_SUITE 
value for OpenCensus suite configuration should be IgniteOpenCensusSuite 
but the current value is OpenCensusMetricExporterSpiTest [1].


Unless this is done on purpose, can anyone help me to fix this issue?

[1] - 
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_OpenCensus=buildTypeSettings_IgniteTests24Java8=%3Cdefault%3E

Re: [DISCUSSION] Maintenance Mode feature

[Sergey Chugunov]

Vladislav, Ivan,

Thank you for your questions and suggestions. Let me answer them.

Vladislav,

If I understood you correctly, you're talking about a node performing some
automatic actions to fix the problem and then join the cluster as usual.

However the original ticket [1] where we faced the need for Maintenance
Mode is about exactly the opposite: avoid doing automatic actions and give
a user the ability to decide what to do.

Also the idea of Maintenance Mode is that the node is able to accept
commands, expose metrics and so on, thus we need all components to be
initialized (some of them may be partially initialized due to their own
maintenance).
To achieve that we need to go through a full cycle of node initialization
including discovery initialization. When discovery is initialized (in
special isolated mode) I don't think it is easy to switch back to normal
operations without a restart.

Ivan,

I think of a node in MM as an almost normal one (maybe with some components
skipped some steps of their initialization). Commands are accepted,
appropriate metrics are exposed e.g. through JMX API and so on.

So as I see it we'll have special commands for control.{sh|bat} CLI
allowing user to see reasons why node switched to maintenance mode and/or
trigger actions to fix the problem (I'm still thinking about proper design
of these actions though).

Of course the user should also be able to fix the problem manually e.g. by
manually deleting corrupted PDS files when node is down. Ideally
Maintenance Mode should be smart enough to figure that out and switch to
normal operations without a restart but I'm not sure if it is possible
without invasive changes of our components' lifecycle.
So I believe this model (node truly started in Maintenance Mode and new
commands in control.{sh|bat}) is a good fit for our current APIs and ways
to interact with the node.

Does it sound reasonable to you?

Thank you!

[1] https://issues.apache.org/jira/browse/IGNITE-13366

On Tue, Sep 1, 2020 at 2:07 PM Ivan Pavlukhin  wrote:

> Sergey,
>
> Actually, I missed the point that the discussed mode affects a single
> node but not a whole cluster. Perhaps I mixed terms "mode" and
> "state".
>
> My next thoughts about maintenance routines are about special
> utilities. As far as I remember MySQL provides a bunch of scripts for
> various maintenance purposes. What user interface for maintenance
> tasks execution is assumed? And what do we mean by "starting" a node
> in a maintenance mode? Can we do some routines without "starting"
> (e.g. try to recover PDS or cleanup)?
>
> 2020-08-31 23:41 GMT+03:00, Vladislav Pyatkov :
> > Hi Sergey.
> >
> > As I understand any switching from/to MM possible only through manual
> > restart a node.
> > But in your example that look like a technical actions, that only
> possible
> > in the case.
> > Do you plan to provide a possibility for client where he can make a
> > decision without a manual intervention?
> >
> > For example: Start node and manually agree with an option and after
> > automatically resolve conflict and back to topology as a stable node.
> >
> > On Mon, Aug 31, 2020 at 5:41 PM Sergey Chugunov <
> sergey.chugu...@gmail.com>
> > wrote:
> >
> >> Hello Ivan,
> >>
> >> Thank you for raising the good question, I didn't think of Maintenance
> >> Mode
> >> from that perspective.
> >>
> >> In short, Maintenance Mode isn't related to Cluster States concept.
> >> According to javadoc documentation of ClusterState enum [1] it is solely
> >> about cache operations and to some extent doesn't affect other
> components
> >> of Ignite node.
> >> From APIs perspective putting the methods to manage Cluster State to
> >> IgniteCluster interface doesn't look ideal to me but it is as it is.
> >>
> >> On the other hand Maintenance Mode as I see it will be managed through
> >> different APIs than a ClusterState and this difference definitely will
> be
> >> reflected in the documentation of the feature.
> >>
> >> Ignite node is a complex piece of many components interacting with each
> >> other, they may have different lifecycles and states; states of
> different
> >> components cannot be reduced to the lowest common denominator.
> >>
> >> However if you have an idea of how to call the feature better to let the
> >> user easier distinguish it from other similar features please share it
> >> with
> >> us. Personally I'm very welcome to any suggestions that make design more
> >> intuitive and easy-to-use.
> >>
> >> Thanks!
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cluster/ClusterState.java
> >>
> >> On Mon, Aug 31, 2020 at 12:32 PM Ivan Pavlukhin 
> >> wrote:
> >>
> >> > Hi Sergey,
> >> >
> >> > Thank you for bringing attention to that important subject!
> >> >
> >> > My note here is about one more cluster mode. As far as I know
> >> > currently we already have 3 modes (inactive, read-only, read-write)
> >> > and the subject is about one