Re: Ignite not friendly for Monitoring

[Denis Magda]

Hello Alexey,

Thanks for the detailed input.

Assuming that Ignite supported the suggested events based model, how can it be 
integrated with mentioned tools like DynaTrace or Nagios? Is this all we need?

—
Denis
 
> On Aug 14, 2017, at 5:02 AM, Alexey Kukushkin 
>  wrote:
> 
> Igniters,
> While preparing some Ignite materials for Administrators I found Ignite is 
> not friendly for such a critical DevOps practice as monitoring. 
> TL;DRI think Ignite misses structured descriptions of abnormal events with 
> references to event IDs in the logs not changing as new versions are released.
> MORE DETAILS
> I call an application “monitoring friendly” if it allows DevOps to:
> 1. immediately receive a notification (email, SMS, etc.)
> 2. understand what a problem is without involving developers 
> 3. provide automated recovery action.
> 
> Large enterprises do not implement custom solutions. They usually use tools 
> like DynaTrace, Nagios, SCOM, etc. to monitor all apps in the enterprise 
> consistently. All such tools have similar architecture providing a dashboard 
> showing apps as “green/yellow/red”, and numerous “connectors” to look for 
> events in text logs, ESBs, database tables, etc.
> 
> For each app DevOps build a “health model” - a diagram displaying the app’s 
> “manageable” components and the app boundaries. A “manageable” component is 
> something that can be started/stopped/configured in isolation. “System 
> boundary” is a list of external apps that the monitored app interacts with.
> 
> The main attribute of a manageable component is a list of “operationally 
> significant events”. Those are the events that DevOps can do something with. 
> For example, “failed to connect to cache store” is significant, while “user 
> input validation failed” is not.
> 
> Events shall be as specific as possible so that DevOps do not spend time for 
> further analysis. For example, a “database failure” event is not good. There 
> should be “database connection failure”, “invalid database schema”, “database 
> authentication failure”, etc. events.  
> 
> “Event” is NOT the same as exception occurred in the code. Events identify 
> specific problem from the DevOps point of view. For example, even if 
> “connection to cache store failed” exception might be thrown from several 
> places in the code, that is still the same event. On the other side, even if 
> a SqlServerConnectionTimeout and OracleConnectionTimeout exceptions might be 
> caught in the same place, those are different events since MS SQL Server and 
> Oracle are usually different DevOps groups in large enterprises!
> 
> The operationally significant event IDs must be stable: they must not change 
> from one release to another. This is like a contract between developers and 
> DevOps.
> 
> This should be the developer’s responsibility to publish and maintain a table 
> with attributes:
> 
> - Event ID
> - Severity: Critical (Red) - the system is not operational; Warning (Yellow) 
> - the system is operational but health is degraded; None - just an info.
> - Description: concise but enough for DevOps to act without developer’s help
> - Recovery actions: what DevOps shall do to fix the issue without developer’s 
> help. DevOps might create automated recovery scripts based on this 
> information.
> 
> For example:
> 10100 - Critical - Could not connect to Zookeeper to discovery nodes - 1) 
> Open ignite configuration and find zookeeper connection string 2) Make sure 
> the Zookeeper is running
> 10200 - Warning - Ignite node left the cluster.
> 
> Back to Ignite: it looks to me we do not design for operations as described 
> above. We have no event IDs: our logging is subject to change in new version 
> so that any patterns DevOps might use to detect significant events would stop 
> working after upgrade.
> 
> If I am not the only one how have such concerns then we might open a ticket 
> to address this.
> 
> 
> Best regards, Alexey

Re: Failure to deserialize simple model object

[Valentin Kulichenko]

Cross-posting to dev

Folks,

I'm confused by the issue discussed in this thread.

Here is the scenario:
- Start server node with a cache with POJO store configured. There is one
type declared, read-through enabled.
- Start client node and execute get() for a key that exists in underlying
DB.
- During deserialization on the client, 'Requesting mapping from grid
failed for' exception is thrown.

Specifying the type explicitly in BinaryConfiguration solves the issue, and
I think I understand technical reasons for this. But is this really
expected? Is it possible to fix the issue without requiring to provide this
configuration?

I thought we do not require to provide types in configuration as long as
there is only one platform involved, am I wrong? If yes, we need to
identify scenarios when this documentation is required and document them.

-Val

On Mon, Aug 14, 2017 at 4:23 AM, franck102  wrote:

> My bad, here is the whole project.
>
> Franck ignite-binary-sample.zip
>  n16158/ignite-binary-sample.zip>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Failure-to-deserialize-simple-model-
> object-tp15440p16158.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Control.sh script and cluster activation

[Valentin Kulichenko]

Agree that this is confusing. I think this functionality should be a part
of Visor CLI tool (likely a new command there).

-Val

On Mon, Aug 14, 2017 at 4:21 PM, Denis Magda  wrote:

> Dmitriy,
>
> I see you contributed control.sh script that activates a cluster after a
> restart. Honestly, I’m a bit confused by it:
>
> 1. How to use it? I could find out that there are some of the parameters
> but the ‘help’ is not implemented. Please fix this and provide a
> description for every parameter you introduced.
>
> 2. Why did we decide to create a specific script for that? Why can’t we
> use existing visorcmd script?
>
> 3. Why the script called “control.sh”?
>
> —
> Denis

Control.sh script and cluster activation

[Denis Magda]

Dmitriy,

I see you contributed control.sh script that activates a cluster after a 
restart. Honestly, I’m a bit confused by it:

1. How to use it? I could find out that there are some of the parameters but 
the ‘help’ is not implemented. Please fix this and provide a description for 
every parameter you introduced.

2. Why did we decide to create a specific script for that? Why can’t we use 
existing visorcmd script?

3. Why the script called “control.sh”?

—
Denis

[GitHub] ignite pull request #2442: Ignite 1.8.10

[mcherkasov]

GitHub user mcherkasov opened a pull request:

https://github.com/apache/ignite/pull/2442

Ignite 1.8.10



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-1.8.10

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2442.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2442


commit caa3acb519e98a86def1ead13e2d0f2e59e620b5
Author: devozerov 
Date:   2017-02-14T09:09:10Z

Merge branch 'ignite-1.7.7' into ignite-1.8.3

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheAdapter.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java
#   modules/platforms/dotnet/Apache.Ignite.AspNet/Properties/AssemblyInfo.cs
#   
modules/platforms/dotnet/Apache.Ignite.Benchmarks/Properties/AssemblyInfo.cs
#   
modules/platforms/dotnet/Apache.Ignite.Core.Tests.TestDll/Properties/AssemblyInfo.cs
#   
modules/platforms/dotnet/Apache.Ignite.Core.Tests/Properties/AssemblyInfo.cs
#   modules/platforms/dotnet/Apache.Ignite.Core/Properties/AssemblyInfo.cs
#   modules/platforms/dotnet/Apache.Ignite.Linq/Properties/AssemblyInfo.cs
#   modules/platforms/dotnet/Apache.Ignite/Properties/AssemblyInfo.cs
#   
modules/platforms/dotnet/examples/Apache.Ignite.Examples/Properties/AssemblyInfo.cs
#   
modules/platforms/dotnet/examples/Apache.Ignite.ExamplesDll/Properties/AssemblyInfo.cs

commit 8874f99f44dc2edf08a525619edb49d5db70b938
Author: dkarachentsev 
Date:   2017-02-14T15:44:57Z

IGNITE-4641 - Refresh client attributes during reconnect

commit ca5956bedfc0c3bd18290c64b6a6c2e3f114a440
Author: Andrey Novikov 
Date:   2017-02-14T16:28:06Z

IGNITE-4472 UI fix, minor fixes

(cherry picked from commit 79e1e53)

commit db9252c9588c671279664484bb8c397312d265e6
Author: Andrey Novikov 
Date:   2017-02-15T09:08:57Z

IGNITE-4678 Node version in range.

(cherry picked from commit 2cfd55d)

commit 58c0c49d31605bf4608e7ee97099b75b324a782f
Author: Andrey Novikov 
Date:   2017-02-16T03:41:30Z

IGNITE-4472 Fixed became this user.

(cherry picked from commit ee832e4)

commit 2ccbf32ea3ecaff1068832accf37235a32734b4b
Author: Andrey Novikov 
Date:   2017-02-16T07:22:22Z

IGNITE-4472 Minor UI fix.

(cherry picked from commit 97c7ed7)

commit ef4886d5be7c708b917e97b1cd5fd766b2ad8450
Author: Andrey Novikov 
Date:   2017-02-16T11:38:40Z

IGNITE-4472 Minor UI fix.

(cherry picked from commit d4efbf3)

commit 05788b3188b30b5a3b39a75fe66301e03658408f
Author: Andrey V. Mashenkov 
Date:   2017-02-17T09:14:53Z

IGNITE-3429: Added BinaryResolver configuration samples for  
org.hibernate.cache.spi.CacheKey. This closes #1516.

commit 1f881aa70a3894af01135f4cc5e341a8130462c2
Author: dkarachentsev 
Date:   2017-02-17T09:34:41Z

IGNITE-4147 - Throw exception on connecting node to cluster with different 
SSL configuration

commit 382fbc9d0b55f794eb4a9045fe72ca06b480062f
Author: dkarachentsev 
Date:   2017-02-17T09:35:18Z

Merge remote-tracking branch 'origin/ignite-1.7.8' into ignite-1.7.8

commit 11bbec487dc174fac1acd6b50c940840305bc75a
Author: Alexey Kuznetsov 
Date:   2017-02-17T10:57:50Z

IGNITE-4436 API for collecting list of running queries and cancel them.
(cherry picked from commit 49237343d53ee33d44e5599cd7fe7da868ee30a1)

commit 2f57760dbb4fba948cd035498d2c7f71869c0665
Author: Andrey V. Mashenkov 
Date:   2017-02-17T13:15:31Z

IGNITE-4624: Scan query optimization. This closes #1509.

commit c0e2df26f056cd11690d821146f05e3fd938906e
Author: dkarachentsev 
Date:   2017-02-20T08:17:35Z

IGNITE-3429 - Rollback due to broken compilation

commit c849534df6583043d6a1d4f454cb981a20896d1a
Author: Andrey Novikov 
Date:   2017-02-21T03:08:55Z

IGNITE-4472 Added user activities in Web Console.

(cherry picked from commit 26ee9c2865648118da97ee8ef84df990359edb96)

commit 9df5e94d5cf14ddd55e29b81989177a7798f7e1a
Author: dkarachentsev 
Date:   2017-02-21T12:34:59Z

IGNITE-4671 - FairAffinityFunction fails on node restart with backupFilter 
set and no backups

commit 9fcb3e74f91c8497b7b1358cdff40950cdf5c568
Author: dkarachentsev 
Date:   2017-02-28T13:05:06Z

IGNITE-4740 - Fix. Service could be deployed/undeployed twice on concurrent 
cancel and discovery 

Re: Request for contributor permission

[Denis Magda]

Hello Zhang Yuan,

Welcome to the Ignite community!

Added you to the list. Below you can find useful information.

Get familiar with Ignite development process described here:
https://cwiki.apache.org/confluence/display/IGNITE/Development+Process

Instructions on how to contribute can be found here:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

Project setup in Intellij IDEA:
https://cwiki.apache.org/confluence/display/IGNITE/Project+Setup

Once you got familiar and were able to run a few examples, you should pick
a Jira ticket you would like to start on. Send an email to the dev list sharing 
your JIRA id, so we can add you as a contributor in Jira.

These are the easy tickets to start with:
https://issues.apache.org/jira/browse/IGNITE-4549?jql=project%20%3D%20IGNITE%20AND%20labels%20in%20(newbie)%20and%20status%20%3D%20OPEN

While those are more advanced but appealing:
https://ignite.apache.org/community/contribute.html#pick-ticket

Looking forward to your contributions!

—
Denis
> On Aug 12, 2017, at 4:30 AM, 张源  wrote:
> 
> Dear Ignite Team,
> 
>   I found an easy ticket to get start up with ignite, so I request for 
> contributors permission.
>   My username is shia.
>   Thanks a lot.
> 
> Zhang Yuan

Re: Hello

[Denis Magda]

Hello,

Welcome to the Ignite community!

Added you to the list. Below you can find useful information.

Get familiar with Ignite development process described here:
https://cwiki.apache.org/confluence/display/IGNITE/Development+Process

Instructions on how to contribute can be found here:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

Project setup in Intellij IDEA:
https://cwiki.apache.org/confluence/display/IGNITE/Project+Setup

Once you got familiar and were able to run a few examples, you should pick
a Jira ticket you would like to start on. Send an email to the dev list sharing 
your JIRA id, so we can add you as a contributor in Jira.

These are the easy tickets to start with:
https://issues.apache.org/jira/browse/IGNITE-4549?jql=project%20%3D%20IGNITE%20AND%20labels%20in%20(newbie)%20and%20status%20%3D%20OPEN

While those are more advanced but appealing:
https://ignite.apache.org/community/contribute.html#pick-ticket

Looking forward to your contributions!

—
Denis

> On Aug 14, 2017, at 8:18 AM, ign...@alexzaitzev.pro wrote:
> 
> 
> Hi all, I am new here. My jira login: alexzaitzev.
> 
> Please, give me access rights to contribute to Ignite. I hope I can help 
> irreparably improve this product.

[jira] [Created] (IGNITE-6059) Use any distributed matrix in K-Means

[Yury Babak (JIRA)]

Yury Babak created IGNITE-6059:
--

 Summary: Use any distributed matrix in K-Means
 Key: IGNITE-6059
 URL: https://issues.apache.org/jira/browse/IGNITE-6059
 Project: Ignite
  Issue Type: Improvement
  Components: ml
Reporter: Yury Babak
 Fix For: 2.2


Currently k-means work only with row/col matrix.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6058) Test fail testTransformResourceInjection broken

[Dmitriy Govorukhin (JIRA)]

Dmitriy Govorukhin created IGNITE-6058:
--

 Summary: Test fail testTransformResourceInjection broken
 Key: IGNITE-6058
 URL: https://issues.apache.org/jira/browse/IGNITE-6058
 Project: Ignite
  Issue Type: Test
Reporter: Dmitriy Govorukhin
 Fix For: 2.2






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2441: IGNITE-6033 Add sorted and multithreaded modes in...

[glukos]

GitHub user glukos opened a pull request:

https://github.com/apache/ignite/pull/2441

IGNITE-6033 Add sorted and multithreaded modes in checkpoint algorithm



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6033

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2441.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2441


commit f773c636b2fa2d59dec6f054d7c9af7b4bfee581
Author: Ivan Rakov 
Date:   2017-08-14T15:54:26Z

IGNITE-6033 Add sorted and multithreaded modes in checkpoint algorithm




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Hello

[ignite]

Hi all, I am new here. My jira login: alexzaitzev.

Please, give me access rights to contribute to Ignite. I hope I can help 
irreparably improve this product.

Re: Batch service deployment

[Dmitriy Setrakyan]

Hi Denis, I agree, we should have an API for batch service deployment. My
comments are inline...

On Mon, Aug 14, 2017 at 2:22 AM, Denis Mekhanikov 
wrote:

> Hi Igniters!
>
> Currently Ignite doesn't have support for batch service deployment, but it
> may be a very useful feature in case of a big number of nodes in a cluster
> and services to be deployed. Each deployment includes write into an
> internal transactional cache, which is the longest part of the procedure.
>
> I propose to optimize it by performing multiple writes in a single
> transaction. It implies an introduction of a few new methods in
> IgniteServices interface.
> I am thinking about the following signatures:
>
>   void deployAll(Iterable cfgs) throws
> IgniteException;
>   IgniteFuture deployAllAsync(Iterable
> cfgs) throws IgniteException;
>
> I'd like to know your opinion on the following questions:
>
>- Do you agree with the proposed signatures?
>

Yes, but Iterable should be changed to Collection to be consistent with
other similar APIs in Ignite.


>- What should happen in case of a failure (some of the configurations
>don't pass validation, or a service with specified name but different
>configuration already exists)? Should partial deployments be performed
> in
>case when some of them fail?
>

Yes, we should allow partial deployment. The exception thrown should have a
collection of services that have failed deployment. It looks like you will
need to create ServiceDeploymentException (extends IgniteException) to
handle this case (in which case, you have to make sure that other deploy
methods also throw it).


>
> Regarding the second question I think that we shouldn't deploy any services
> in a batch if we encounter any problems with some of them.
>
> Also cancelAll method may be optimized in a similar way, but no interface
> changes are needed there.
>
> Ticket: https://issues.apache.org/jira/browse/IGNITE-5145
>
> --
> Cheers,
> Denis Mekhanikov
>

[GitHub] ignite pull request #2439: IGNITE-6052 request cluster state from daemon nod...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2439


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6057) SQL: Full scan should be performed through data pages bypassing primary index

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6057:
---

 Summary: SQL: Full scan should be performed through data pages 
bypassing primary index
 Key: IGNITE-6057
 URL: https://issues.apache.org/jira/browse/IGNITE-6057
 Project: Ignite
  Issue Type: Improvement
  Components: persistence, sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov
 Fix For: 2.2


Currently both SQL full scan and {{CREATE INDEX}} commands iterate through 
primary index to get all existing values. Consider that we have 10 entries per 
data page on average. In this case we will have to read the same data page 10 
times when reaching relevant keys in different parts of index tree. This could 
be very inefficient on certain workloads.

We should iterate over data pages directly instead. This way a page with 10 
entries will be accessed only once. However, we should take cache groups in 
count - if there are too many entries from other logical caches, this approach 
could make situation even worse, unless we have a mechanism to skip unnecessary 
entries (or the whole pages!) efficiently.

Probably we should develop a cost-based model, which will take in count the 
following statistics:
1) Average entry size. The longer the entry, the lesser the benefit. Especially 
if overflow pages are used frequently. 
2) Cache groups. Ideally, we should estimate number of entries from all logical 
caches. The more entries from other caches, the lesser the benefit.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6056) Grid hangs on partition map exhcange during failover test with 500 logical and 26 physical caches

[Ksenia Rybakova (JIRA)]

Ksenia Rybakova created IGNITE-6056:
---

 Summary: Grid hangs on partition map exhcange during failover test 
with 500 logical and 26 physical caches
 Key: IGNITE-6056
 URL: https://issues.apache.org/jira/browse/IGNITE-6056
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Ksenia Rybakova


Grid hangs on partition map exhcange during failover test with 500 logical and 
26 physical caches.
Load config:
- CacheRandomOperationBenchmark
- 8 client nodes, 40 server nodes
- 40K perloading, 60K key range
- 26 physical caches
- 500 logical caches
- 2 backups
- 1 server node is being restarted periodically
Complete configs are attached.

Preloading phase is ok, then main test starts and after some time grid hangs.






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6055) SQL: Add String length constraint

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6055:
---

 Summary: SQL: Add String length constraint
 Key: IGNITE-6055
 URL: https://issues.apache.org/jira/browse/IGNITE-6055
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov
 Fix For: 2.2


We should support {{CHAR(X)}} and {{VARCHAR{X}} syntax. Currently, we ignore 
it. First, it affects semantics. E.g., one can insert a string with greater 
length into a cache/table without any problems. Second, it limits efficiency of 
our default configuration. E.g., index inline cannot be applied to {{String}} 
data type as we cannot guess it's length.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6054) SQL: Add option to store primitive keys in plain form for CREATE TABLE

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6054:
---

 Summary: SQL: Add option to store primitive keys in plain form for 
CREATE TABLE
 Key: IGNITE-6054
 URL: https://issues.apache.org/jira/browse/IGNITE-6054
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov
 Fix For: 2.2


Currently we create separate internal type for primary key columns. This is 
necessary to avoid clashes between keys of the same type within the same caches 
(ironically, we do not allow multiple dynamic tables per cache).

The most widely used PK is single-column key of {{Long}} or {{String}} data 
type. If we store a key plain {{long}}, it will consume 9 bytes. If we store it 
as an object with long field, it will consume 24 + 9 = 33 bytes. What is worse, 
in the latter case we will have to copy key object back and forth between page 
memory and application code many times, while for plain long key we simply do 
{{Unsafe.getLong}}.

For this reason, it makes sense to introduce special mode for {{CREATE TABLE}} 
command, when key will not be wrapped into a class, and will be stored as is. 
Let's name it {{plainPrimaryKey}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2347: IGNITE-5843 Ignite PDS: We don't save cache confi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2347


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6053) IgniteCache.clear clears local cache with same names on all server nodes

[Evgenii Zhuravlev (JIRA)]

Evgenii Zhuravlev created IGNITE-6053:
-

 Summary: IgniteCache.clear clears local cache with same names on 
all server nodes
 Key: IGNITE-6053
 URL: https://issues.apache.org/jira/browse/IGNITE-6053
 Project: Ignite
  Issue Type: Bug
Affects Versions: 1.7
Reporter: Evgenii Zhuravlev
 Fix For: 2.2


Clear on LOCAL cache should clear the only cache on the current node, not on 
all nodes, that have local caches with same names.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2440: IGNITE-5416: add Kafka-specific builder for Strea...

[endian675]

GitHub user endian675 opened a pull request:

https://github.com/apache/ignite/pull/2440

IGNITE-5416: add Kafka-specific builder for StreamSingleTupleExtractor



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/endian675/ignite ignite-5416-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2440.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2440


commit 42a1f35bb729f8b1bf145d7594071fa67fc74857
Author: Michael Griggs 
Date:   2017-08-14T13:32:51Z

IGNITE-5416: add Kafka-specific builder for StreamSingleTupleExtractor




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2329: IGNITE-5741

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2329


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: SQL usability issues

[Dmitriy Setrakyan]

On Mon, Aug 14, 2017 at 2:52 AM, Igor Sapego  wrote:

> Dmitriy,
>
> But what would we put in such generic section?
>

- JAR in Ignite that contains JDBC driver
- connection string
- port (with explanation how to change it)
- username/password with explanation why Ignite does not require it.
- screenshot of configuring these properties with a popular SQL tool


>
> Best Regards,
> Igor
>
> On Sun, Aug 13, 2017 at 7:20 PM, Dmitriy Setrakyan 
> wrote:
>
> > On Fri, Aug 11, 2017 at 7:31 PM, Denis Magda 
> wrote:
> >
> > > Dmitriy,
> > >
> > > That's the documentation that shows how to configure Pentaho via JDBC:
> > > https://apacheignite-tools.readme.io/v2.1/docs/pentaho
> >
> >
> > I am not sure I agree. Why would I look at Pentaho integration when
> trying
> > to configure a generic SQL viewer tool? We need a generic documentation
> > section for configuring JDBC/ODBC drivers with 3rd party tools.
> >
> >
> > >
> > >
> > > Plus on the tools' domain you can see Tableau based ODBC example.
> > >
> > > So, I'm not sure we need something else from the documentation
> > standpoint.
> > > I would rather add a direct reference from JDBC/ODBC docs to Pentaho
> and
> > > Tablea.
> > >
> > > Denis
> > >
> > > On Friday, August 11, 2017, Dmitriy Setrakyan 
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > I have tried to connect to Ignite from DBeaver [1], and realized that
> > > there
> > > > are some usability issues we need to address before the next release:
> > > >
> > > >1. We need to have documentation on how to configure JDBC and ODBC
> > > >drivers with external SQL tools [2]
> > > >2. You cannot highlight multiple SQL statements and run them
> > together
> > > > [3]
> > > >3. Commands like *DESCRIBE* or *SHOW* do not work [4]
> > > >4. Schema, index, and table metadata is not displayed [5]. Looks
> > like
> > > >this fix was already implemented.
> > > >
> > > > The links to the tickets are below.
> > > >
> > > > [1] http://dbeaver.jkiss.org/
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-6048
> > > > [3] https://issues.apache.org/jira/browse/IGNITE-6046
> > > > [4] https://issues.apache.org/jira/browse/IGNITE-6047
> > > > [5] https://issues.apache.org/jira/browse/IGNITE-5233
> > > >
> > > > D.
> > > >
> > >
> >
>

Ignite not friendly for Monitoring

[Alexey Kukushkin]

Igniters,
While preparing some Ignite materials for Administrators I found Ignite is not 
friendly for such a critical DevOps practice as monitoring. 
TL;DRI think Ignite misses structured descriptions of abnormal events with 
references to event IDs in the logs not changing as new versions are released.
MORE DETAILS
I call an application “monitoring friendly” if it allows DevOps to:
1. immediately receive a notification (email, SMS, etc.)
2. understand what a problem is without involving developers 
3. provide automated recovery action.

Large enterprises do not implement custom solutions. They usually use tools 
like DynaTrace, Nagios, SCOM, etc. to monitor all apps in the enterprise 
consistently. All such tools have similar architecture providing a dashboard 
showing apps as “green/yellow/red”, and numerous “connectors” to look for 
events in text logs, ESBs, database tables, etc.

For each app DevOps build a “health model” - a diagram displaying the app’s 
“manageable” components and the app boundaries. A “manageable” component is 
something that can be started/stopped/configured in isolation. “System 
boundary” is a list of external apps that the monitored app interacts with.

The main attribute of a manageable component is a list of “operationally 
significant events”. Those are the events that DevOps can do something with. 
For example, “failed to connect to cache store” is significant, while “user 
input validation failed” is not.

Events shall be as specific as possible so that DevOps do not spend time for 
further analysis. For example, a “database failure” event is not good. There 
should be “database connection failure”, “invalid database schema”, “database 
authentication failure”, etc. events.  

“Event” is NOT the same as exception occurred in the code. Events identify 
specific problem from the DevOps point of view. For example, even if 
“connection to cache store failed” exception might be thrown from several 
places in the code, that is still the same event. On the other side, even if a 
SqlServerConnectionTimeout and OracleConnectionTimeout exceptions might be 
caught in the same place, those are different events since MS SQL Server and 
Oracle are usually different DevOps groups in large enterprises!

The operationally significant event IDs must be stable: they must not change 
from one release to another. This is like a contract between developers and 
DevOps.

This should be the developer’s responsibility to publish and maintain a table 
with attributes:
 
- Event ID
- Severity: Critical (Red) - the system is not operational; Warning (Yellow) - 
the system is operational but health is degraded; None - just an info.
- Description: concise but enough for DevOps to act without developer’s help
- Recovery actions: what DevOps shall do to fix the issue without developer’s 
help. DevOps might create automated recovery scripts based on this information.

For example:
10100 - Critical - Could not connect to Zookeeper to discovery nodes - 1) Open 
ignite configuration and find zookeeper connection string 2) Make sure the 
Zookeeper is running
10200 - Warning - Ignite node left the cluster.

Back to Ignite: it looks to me we do not design for operations as described 
above. We have no event IDs: our logging is subject to change in new version so 
that any patterns DevOps might use to detect significant events would stop 
working after upgrade.

If I am not the only one how have such concerns then we might open a ticket to 
address this.


Best regards, Alexey

[jira] [Created] (IGNITE-6052) Check cluster state from daemon node return incorrect cluster state

[Dmitriy Govorukhin (JIRA)]

Dmitriy Govorukhin created IGNITE-6052:
--

 Summary: Check cluster state from daemon node return incorrect 
cluster state
 Key: IGNITE-6052
 URL: https://issues.apache.org/jira/browse/IGNITE-6052
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.0
Reporter: Dmitriy Govorukhin
Assignee: Dmitriy Govorukhin
Priority: Critical
 Fix For: 2.2


Daemon node must requested cluster state via compute grid from server node.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6051) Improve future listeners model in DataStreamerImpl

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-6051:


 Summary: Improve future listeners model in DataStreamerImpl
 Key: IGNITE-6051
 URL: https://issues.apache.org/jira/browse/IGNITE-6051
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.1
Reporter: Igor Seliverstov
Assignee: Igor Seliverstov


Improve future listeners model on update not to add the same listener to a 
future more than once ({{DataStreamerImpl.java:1410}})



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2428: IGNITE-4991 Do not print out system properties wh...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2428


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2386: IGNITE-5890 Estimated time to rebalance completio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2386


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2438: IGNITE-4172 SQL: Add support for Java 8 Time API ...

[asfedotov]

GitHub user asfedotov opened a pull request:

https://github.com/apache/ignite/pull/2438

IGNITE-4172 SQL: Add support for Java 8 Time API classes in date\time 
funcitons

For CI purposes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4172

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2438.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2438


commit 84c7427a53a8e1712b1d0b763d7539c9cb844cb6
Author: Sergey Stronchinskiy 
Date:   2017-07-04T11:51:25Z

IGNITE-5532 .NET: Split CacheLinqTest into partial classes

This closes #2226

commit 64c156e9252395504af00f09d934f36b6bc21913
Author: Igor Sapego 
Date:   2017-07-04T16:42:33Z

IGNITE-5663: ODBC: Closing cursor do not reset prepared statement anymore

commit 80c95ff79f344daf1fca3f094733a24bac2a218d
Author: Igor Sapego 
Date:   2017-07-05T15:41:28Z

IGNITE-5576: Added Compute::Run() for C++

commit 836906c89dfb880ac602046c37b3a2dba3ebdc46
Author: samaitra 
Date:   2017-07-06T04:28:15Z

IGNITE-5695 FlinkIgniteSinkSelfTest is failing due to conflicting default 
test timeout and default flush frequency - Fixes #2241.

Signed-off-by: samaitra 

commit 651ffc544bbc32cded55626adcd3ed4cc74f11ce
Author: shroman 
Date:   2017-07-06T05:00:08Z

Removed unnecessary line from the comments.

commit d1d6802378d874b039f775fe787f78c507661bb2
Author: devozerov 
Date:   2017-07-07T09:36:13Z

Merge branch 'ignite-2.1'

commit 45cd87fe73db117f5148ed2006f8de8d2517bbfe
Author: mcherkasov 
Date:   2017-06-30T17:23:55Z

IGNITE-5554 ServiceProcessor may process failed reassignments in timeout 
thread

commit fa974286e8f066a8d6aa57519edf5ec7761be095
Author: Igor Sapego 
Date:   2017-07-07T13:49:15Z

IGNITE-5582: Implemented Compute::Broadcast for C++

commit 01f504ff83cc77f80d37981b5c5a15b653861bbd
Author: NSAmelchev 
Date:   2017-07-10T12:03:01Z

IGNITE-5087 Enum comparison fails after marshal-unmarshal with 
BinaryMarshaller.

commit ecfbc2c97464ad7da3f24afaaf1868a2d2fdb87e
Author: devozerov 
Date:   2017-07-11T09:17:41Z

Merge branch 'ignite-2.1'

# Conflicts:
#   
modules/platforms/dotnet/Apache.Ignite.Core.Tests/Apache.Ignite.Core.Tests.csproj

commit 1be9b40c37efcbf332ebeeefc865c2fe864339e7
Author: sboikov 
Date:   2017-07-11T09:42:54Z

Exchange future cleanup, added special tasks for reassign/force rebalance.

commit 8d4a0c2ca2abc17a1d54fa0d33b161531fa59b12
Author: Pavel Tupitsyn 
Date:   2017-07-11T09:49:16Z

Merge branch 'ignite-2.1'

commit bf25b5c52be044f07076c0800447230c75174db3
Author: Slava Koptilin 
Date:   2017-07-07T12:35:33Z

ignite-5562: assert statements were changed to the 'if' blocks

commit e93b28488693381fcd232de93087ab8ec1d0f5bb
Author: sboikov 
Date:   2017-07-11T11:18:52Z

ignite-5446 Only lateAffinity logic in CacheAffinitySharedManager.

commit 5c363184c80f2fd8b79f1075d1eacbf9af5369a1
Author: Denis Magda 
Date:   2017-07-11T19:20:16Z

Simplified Memory Policies Example

commit b95f76f8a0a3a7e920f78f20b3d814112fc6d522
Author: sboikov 
Date:   2017-07-12T05:47:04Z

ignite-5727 Call TcpCommunicationSpi's discovery listener first

commit 120384fca2b5f6f141207697f776d7859afa857f
Author: devozerov 
Date:   2017-07-12T06:48:51Z

Merge branch 'ignite-2.1'

commit 5394bbdeff4e9fb97d3b413bf30001ede580bdd7
Author: sboikov 
Date:   2017-07-13T10:30:59Z

Unnecessary synchronous rollback in GridDhtTxLocal.prepareAsync

commit 00c6b6c4ba00fa6577f74fc95b378737fb5a789c
Author: Alexander Menshikov 
Date:   2017-07-13T12:24:59Z

IGNITE-5567 Make benchmark Ignite.reentrantLock vs IgniteCache.lock

commit 18bdfe96a1e579371108c661e3374183c58a296d
Author: Alexey Goncharuk 
Date:   2017-07-13T12:42:30Z

Fixed NPE in tests

commit 7338445ac9c1a2343fd41cdd20785de07b727796
Author: dkarachentsev 
Date:   2017-07-13T13:00:08Z

IGNITE-5597 - Fix javadoc in Affinity and AffinityFunction for REPLICATED 
cache. This closes #2268.

commit d9ed07c67e4a4ff3a9de543cbe039ac2a48f03a0
Author: Sergey Chugunov 
Date:   2017-07-13T14:32:06Z

Functionality of muted test is debated now

commit 871d9260f3b32bed5273852dbdb74c758f73d383
Author: Sergey Chugunov 
Date:   

[jira] [Created] (IGNITE-6050) Eternal wait in DataStreamerTest.TestBufferSize()

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-6050:


 Summary: Eternal wait in DataStreamerTest.TestBufferSize()
 Key: IGNITE-6050
 URL: https://issues.apache.org/jira/browse/IGNITE-6050
 Project: Ignite
  Issue Type: Test
  Components: platforms
Affects Versions: 2.1
Reporter: Igor Seliverstov


There are several places where Future.Wait() is used without timeout. This 
leads tests failures on timeout and makes difficult to determine the cause of 
the issue



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Batch service deployment

[Denis Mekhanikov]

Hi Igniters!

Currently Ignite doesn't have support for batch service deployment, but it
may be a very useful feature in case of a big number of nodes in a cluster
and services to be deployed. Each deployment includes write into an
internal transactional cache, which is the longest part of the procedure.

I propose to optimize it by performing multiple writes in a single
transaction. It implies an introduction of a few new methods in
IgniteServices interface.
I am thinking about the following signatures:

  void deployAll(Iterable cfgs) throws IgniteException;
  IgniteFuture deployAllAsync(Iterable
cfgs) throws IgniteException;

I'd like to know your opinion on the following questions:

   - Do you agree with the proposed signatures?
   - What should happen in case of a failure (some of the configurations
   don't pass validation, or a service with specified name but different
   configuration already exists)? Should partial deployments be performed in
   case when some of them fail?

Regarding the second question I think that we shouldn't deploy any services
in a batch if we encounter any problems with some of them.

Also cancelAll method may be optimized in a similar way, but no interface
changes are needed there.

Ticket: https://issues.apache.org/jira/browse/IGNITE-5145

-- 
Cheers,
Denis Mekhanikov

[jira] [Created] (IGNITE-6049) Try to cache DataStreamer receiver

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-6049:


 Summary: Try to cache DataStreamer receiver
 Key: IGNITE-6049
 URL: https://issues.apache.org/jira/browse/IGNITE-6049
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.1
Reporter: Igor Seliverstov


Currently we send serialized DataStreamer receiver ({{StreamReceiver}}) in each 
{{DataStreamerRequest}}. It makes overhead on message deserialization time.

Also there is a default receiver, which is used in most of cases. As far as I 
understand It isn't necessary to send it since this instance is stateless.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2424: IGNITE-6016 Get rid of checking topology hash in ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2424


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2437: IGNITE-5991

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/2437

IGNITE-5991



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5991

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2437.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2437


commit d38ef59ac299b7c2231cdd4c71ce9359c74b338c
Author: devozerov 
Date:   2017-08-08T12:49:19Z

Created example as a starting point.

commit c440314307980dcaf0944b2c826458e0888ed4e7
Author: devozerov 
Date:   2017-08-08T12:51:25Z

Streaming flag on API.

commit 3cd9c8958d3707d429a2dd5bf13656ef33d8f4b7
Author: devozerov 
Date:   2017-08-08T13:56:28Z

Passed streaming flag to query executor.

commit 8efa03604931b0bfcc958dc3d4b6fe318a8b6d5c
Author: devozerov 
Date:   2017-08-09T08:55:29Z

Merge branch 'master' into ignite-5991

commit 3a4c2d75c61a64accd9059c5cecf46267cff8b4e
Author: devozerov 
Date:   2017-08-09T09:38:04Z

Consuming results into own target.

commit d2c86a7c9d0c2db0bc1757a2004a8c614c874862
Author: devozerov 
Date:   2017-08-09T12:46:13Z

Minors.

commit e000fb08f50c1bd5154c96e14ffc5070da1d147d
Author: devozerov 
Date:   2017-08-09T12:48:50Z

Started working on result target.

commit 815178d6a57ff2e13003a5be12d8630537613c4d
Author: devozerov 
Date:   2017-08-09T15:09:24Z

WIP on target.

commit 9f88f51555264323a610eb0cc12806de4ac3e425
Author: devozerov 
Date:   2017-08-10T10:31:26Z

Merge branch 'master' into ignite-5991

commit 0b88846c8a9cc5abebee630d1aac9df9fb109cee
Author: devozerov 
Date:   2017-08-10T11:36:40Z

WIP.

commit 34fa8a08a8e25db6e6cb0a41a515517c3a062fba
Author: devozerov 
Date:   2017-08-10T12:53:52Z

WIP.

commit 8fcc88009e2cb49b0b03f186a6f8409e9410ef5a
Author: devozerov 
Date:   2017-08-10T12:56:22Z

WIP on flags.

commit 3bc3e4bc87d3b602c0288e1541a8e52633d7d104
Author: devozerov 
Date:   2017-08-10T12:57:58Z

Better flag naming.

commit f7afc0dc7ae3954aaf55d5ef1fa6e28afc974011
Author: devozerov 
Date:   2017-08-10T13:23:08Z

Added last page to the response.

commit d465d11ce0efbe35a58e3e78e2cb5895b9b80444
Author: devozerov 
Date:   2017-08-10T14:04:08Z

Done.

commit d1fdb746c9be80072a50c3de5534215e7ad53983
Author: devozerov 
Date:   2017-08-10T14:13:11Z

Done.

commit 00ccadb0c8c157e1c03a0d57c4751b4d85ff322c
Author: devozerov 
Date:   2017-08-10T14:14:25Z

Minors.

commit 9dc947d8c5a0cec22890a1dde86d91cd1ceea685
Author: devozerov 
Date:   2017-08-10T14:17:19Z

Merge branch 'ignite-6027' into ignite-5991

commit c7c122aee8c379ed0f0240e1bde34c345ce6d14a
Author: devozerov 
Date:   2017-08-10T14:25:16Z

Removed unused argument.

commit c955d18ebe7a6f4d567a9719301f3c9b434d6e17
Author: devozerov 
Date:   2017-08-10T14:41:06Z

Identified places for separate threads.

commit 5f121e0e14d3e3c0b634851164cb9a3ce89d6d54
Author: devozerov 
Date:   2017-08-10T14:45:27Z

Propagated node ID to node results.

commit d50edfd55c6b29411624bb8d8f33b0b89272b1de
Author: devozerov 
Date:   2017-08-10T14:57:30Z

Minors.

commit 284387b14ecdb67225e597bdd822d02fed0d3ef0
Author: devozerov 
Date:   2017-08-14T06:55:34Z

Merge branch 'master' into ignite-5991

commit 40e45ad7663efdf59138239e2fd24a5f001a0e5b
Author: devozerov 
Date:   2017-08-14T07:15:31Z

Lazy worker.

commit 8c4527371b884b8244ecbf23aab1051a418eb69a
Author: devozerov 
Date:   2017-08-14T07:34:02Z

Lazy worker key.

commit ccdd179a8563c7585104982c536402b18e5ef367
Author: devozerov 
Date:   2017-08-14T07:48:07Z

Implemented "onQueryRequest" handling.

commit f9960dff9d0e51b5a9b32473e5970d5d6b06a6a6
Author: devozerov 
Date:   2017-08-14T07:50:25Z

Implemented "onQueryRequest" handling.

commit 9e119bddf900f38b47bd1a1a64d5290c56d15d35
Author: devozerov 
Date:   2017-08-14T07:54:14Z

Next page request.

commit 4f1421cdbd1f5e48375eb97c8635cfec8c9e17b6
Author: devozerov 
Date:   2017-08-14T07:54:22Z

Next page request.

commit a063e21fa6a66326bb9a3bd15af81cf5178aafd0
Author: devozerov 
Date:   2017-08-14T07:57:22Z

Minors.




---
If your project is 

[GitHub] ignite pull request #2408: IGNITE-5941 : Fixed index name length restriction...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2408


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2436: IGNITE-5439 JDBC thin: support query cancel

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/2436

IGNITE-5439  JDBC thin: support query cancel



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5439

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2436.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2436


commit 7c294e473c3efec5956b04c2a02bcfc941f7e4c1
Author: tledkov-gridgain 
Date:   2017-08-08T16:42:34Z

IGNITE-5439 JDBC thin: support query cancel

commit f5152170ffc75d550db0e8da0b3bf60acc9cd1a7
Author: tledkov-gridgain 
Date:   2017-08-14T08:15:02Z

Merge branch '_master' into ignite-5439




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---