[jira] [Created] (IGNITE-9623) AmbiguousMatchException Introduced by New .NET Core 2.1 String.Contains Overloads

[Charles Salmon (JIRA)]

Charles Salmon created IGNITE-9623:
--

 Summary: AmbiguousMatchException Introduced by New .NET Core 2.1 
String.Contains Overloads
 Key: IGNITE-9623
 URL: https://issues.apache.org/jira/browse/IGNITE-9623
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
 Environment: .NET Core 2.1
Reporter: Charles Salmon


A couple of new String.Contains overloads were proposed 
[here|https://github.com/dotnet/corefx/issues/25094], and are documented as 
being included in .NET Core 2.1 
[here|https://docs.microsoft.com/en-us/dotnet/api/system.string.contains?view=netcore-2.1].

The {{MethodVisitor}}, provided as part of {{Apache.Ignite.Linq}} blows up 
while initializing the {{Delegates}} dictionary, as the result of this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4776: Ignite 2.4.9 p1

[antkr]

GitHub user antkr opened a pull request:

https://github.com/apache/ignite/pull/4776

Ignite 2.4.9 p1



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.9-p1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4776.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4776


commit d2659d0ec9f6e1a0b905fc7bf23b65fd5522c80a
Author: Alexander Paschenko 
Date:   2018-03-14T09:23:37Z

IGNITE-7253: JDBC thin driver: implemented streaming. This closes #3499. 
This closes #3591.

commit bc9018ef8b116f81b8e06d2ff7651ba2b6c7beae
Author: tledkov-gridgain 
Date:   2018-03-19T08:01:26Z

IGNITE-7029: JDBC thin driver now supports multiple IP addresses with 
transparent failover. This closes #3585.

commit 587022862fd5bdbb076ab6207ae6fd9bc7583c13
Author: Sergey Chugunov 
Date:   2018-03-16T16:24:17Z

IGNITE-7964 rmvId is stored to MetaStorage metapage during operations - 
Fixes #3645.

commit 006ef4d15e4faedb6dfce6ce9637602055b97293
Author: tledkov-gridgain 
Date:   2018-03-22T11:47:06Z

IGNITE-7436: Simple username/password authentication. This closes #3483.

commit 1c7f59c90514670e802d9d07544b00b7562fe6d2
Author: Pavel Tupitsyn 
Date:   2018-01-30T09:48:16Z

.NET: Fix build status icon in README

commit 162df61b305fccfc55e186d07351727f35b55179
Author: Pavel Tupitsyn 
Date:   2018-02-01T11:53:16Z

IGNITE-7561 .NET: Add IServices.GetDynamicServiceProxy

This closes #3457

commit 9a0328ebbc9d52f8e96898a384fa45743d2efa5b
Author: Pavel Tupitsyn 
Date:   2018-02-02T12:01:27Z

.NET: Update README regarding C++ interop and thin client

commit b804cfea51c87724b45b40de4fd58d300c049be1
Author: Pavel Tupitsyn 
Date:   2018-01-31T09:39:19Z

.NET: Suppress API parity check for SSL in ClientConnectorConfiguration

commit 6f8014de7250c4c0d87cbc8764afae4a225f654b
Author: apopov 
Date:   2018-02-13T10:13:15Z

IGNITE-3111 .NET can be now configured SSL without Spring

This closes #3498

commit 5131bcd71ce787cf2c61bf98446f5ec0a616ab1c
Author: Pavel Tupitsin 
Date:   2018-02-16T20:36:01Z

IGNITE-3111 .NET: Configure SSL without Spring - cleanup

* Remove unused members from ISslContextFactory
* Fix namespaces
* Remove unused files
* Cleanup tests

commit 4ac4645dcf6e85883ce0de46ba1253ba8135804e
Author: Pavel Tupitsyn 
Date:   2018-02-18T20:22:27Z

.NET: Fix LoadDllTest, IgniteStartStopTest

commit 8709785814a432f981c30274a55e2ef730667421
Author: Pavel Tupitsyn 
Date:   2018-02-18T20:27:29Z

.NET: Fix SslConfigurationTest

commit d2a1136fd725b0e09e1e655152509aeae755c368
Author: Pavel Tupitsyn 
Date:   2018-02-26T22:15:11Z

IGNITE-7329 .NET: Thin client: SSL connection implemented

This closes #3447

commit c2a5c5df71197418bd126e87f62e66681ac25b76
Author: Pavel Tupitsin 
Date:   2018-03-05T16:07:32Z

IGNITE-7878 .NET: Ignore TestStartDefault

commit eec0a6714dd9f59edb637c8c068205843f5b9af6
Author: Alexey Popov 
Date:   2018-03-06T17:47:18Z

IGNITE-7889: .NET: LINQ: Fix GroupBy with Where

This closes #3608

commit 74fbd41f7f4e0b129797297c603c471ea558f2f6
Author: Pavel Tupitsyn 
Date:   2018-03-07T08:10:08Z

IGNITE-7566 .NET: Build scripts: stop build when one of the steps fails

commit 4ae42e6d525e04c182c854785b5526617f72ade6
Author: Pavel Tupitsyn 
Date:   2018-03-07T08:25:07Z

IGNITE-7566 .NET: Build scripts: stop build when one of the steps fails

commit 3e334ec43f0090d9feb6574d23955c4ca7a90c14
Author: Sergey Stronchinskiy 
Date:   2018-03-07T08:44:51Z

IGNITE-5298 .NET: DML update via LINQ

This closes #3599

commit 5c90bc77e014da526d64ab9f76b813685c52d6f3
Author: Pavel Tupitsyn 
Date:   2018-03-12T13:21:41Z

IGNITE-7878 .NET: Un-Ignore TestStartDefault

This reverts commit 18c6b4a5d079874e9c156fa39564970ad3c31914.

commit 35ef3bcb4d2ae6ea39203fbceb9e9eb5026eb213
Author: Pavel Tupitsyn 
Date:   2018-03-12T17:12:13Z

.NET: Update DEVNOTES - better .NET Core test commands

commit e9ea2da7d7f6afa609475f16b0250de21e6c139e
Author: Pavel Tupitsyn 
Date:   2018-03-14T21:05:32Z

.NET: Fix code inspection errors

commit 99d022bb3257a2a39e121dba47787e81356a318f
Author: Pavel Tupitsyn 
Date:   2018-03-15T13:48:46Z

.NET: Suppress incorrect code inspection errors

commit e093e51f88cf304e64fcf5535279cc176937fbea
Author: Igor Sapego 
Date:   2018-03-26T10:33:40Z

IGNITE-7852: ODBC: Implemented username/password authentication. This 
closes #3671.

commit 6a4fa6cc87b7d78aed7e5c6a2531d9cfa657bda0
Author: devozerov 
Date:   2018-03-26T10:50:36Z

Merge branch 'ignite-2.4.4-.net' into ignite-2.4.4

commit b630e8fd8d39bed21e044ce55eb6986da0d1f02e
Author: Anton Kalashnikov 
Date:   2018-03-26T11:57:11Z

GG-13638 move setting 

ignite .net plugin for security

[wt]

I have almost completed a plugin in java and was wondering if this plugin
could be made available in .net. I have seen an example online (here -
https://dzone.com/articles/implementing-ignitenet-plugin-distributed-semaphor)
where plugin methods in java are made available in .net. Do you know if it
is possible to implement a security plugin in this fashion or is that level
of integration not possible between java and .net. I can see there are no
security classes available in .net so all the wiring would need to be
developed. would be great to get some of the gridgain developers with .net
knowledge to comment. Perhaps it is planned to be made available in the
future?



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: loadcache very very slow

[wt]

hi

The reason i am asking here and not on the users forum is to get down to the
code details of how loadcache works and specifically what the developers
recommendation is on using it for bulk load operations. In reading up on the
subject, i discovered that it is suggested to override the method and use a
different implementation to the default. Asking on the users forum is only
going to go into details like the env and server and drivers etc. I can tell
you now that i can query the same jdbc driver in beaver on the same database
and get back throughput i would have expected to see with the generated
project. Unfortunately the generated project from the web console is only
useful for basic configs and the loading aspect to it should ultimately
either be removed or updated to support bulk inserts by default. I have lost
confidence in using the web console and generated project due to a series of
issues including this.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: First pull requests

[Dmitriy Pavlov]

Hi David,

Please make sure all tickets are in Patch Available state, so Ignite
committers and contributors can see these tickets in corresponding filters.

Sometimes it is easy to find maintainer, so JIRA mentions work well.
Sometimes when it is not clear, the dev list may be used.

Feel free to remind about tickets in dev list if there is no feedback from
Igniters.

Sincerely,
Dmitriy Pavlov

пн, 17 сент. 2018 г. в 21:17, David Harvey :

> I'm new to this process, and I've created three pull requests, and I'm
> trying to figure out how I can get some eyes to look at them.
>
>- IGNITE-7616  which
>add some missing MXbeans for thread pools.  I've identified some
>contributors that worked on the files before, but when I type [ ~ name
> ] in
>Jira using parts of names found in git or here
>
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers
> ,
>Jira generally doesn't find those people.
>- IGNITE-9026  which
>fixes some bugs with CONTINUOUS peer class loading when such a loaded
> class
>makes a call to another node.   These addressed the problems we had,
> but I
>still have to to write a couple of new tests.   However, it would
> useful to
>get feedback on the approach, because it fundamentally changes the
> search
>process.
>- IGNITE-9365 which adds an optional affinityBackupFilter that can be
>configured in Spring  to separate primary and backup partitions based on
>some node attribute/environment variable, with an example of how to use
>this to force primary and backup to different AWS Availability Zones.
> Val
>said he would look at this one.
>
>
> Thanks,
> Dave Harvey
>

First pull requests

[David Harvey]

I'm new to this process, and I've created three pull requests, and I'm
trying to figure out how I can get some eyes to look at them.

   - IGNITE-7616  which
   add some missing MXbeans for thread pools.  I've identified some
   contributors that worked on the files before, but when I type [ ~ name ] in
   Jira using parts of names found in git or here
   
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers,
   Jira generally doesn't find those people.
   - IGNITE-9026  which
   fixes some bugs with CONTINUOUS peer class loading when such a loaded class
   makes a call to another node.   These addressed the problems we had, but I
   still have to to write a couple of new tests.   However, it would useful to
   get feedback on the approach, because it fundamentally changes the search
   process.
   - IGNITE-9365 which adds an optional affinityBackupFilter that can be
   configured in Spring  to separate primary and backup partitions based on
   some node attribute/environment variable, with an example of how to use
   this to force primary and backup to different AWS Availability Zones.  Val
   said he would look at this one.


Thanks,
Dave Harvey

[GitHub] ignite pull request #4775: Ignite 6587

[agura]

GitHub user agura opened a pull request:

https://github.com/apache/ignite/pull/4775

Ignite 6587



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agura/incubator-ignite ignite-6587

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4775.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4775


commit 3b25cb934838feb3300c7088db4b3840ec7f9ab5
Author: Andrey Kuznetsov 
Date:   2018-05-07T12:43:26Z

IGNITE-6587: Refactored ServerImpl.TcpServer to use GridWorker.

commit 665ba863ceb95b643a805d6920f3f384275dae20
Author: Andrey Kuznetsov 
Date:   2018-05-07T13:06:22Z

IGNITE-6587: Refactored ServerImpl.RingMessageWorker to use GridWorker.

commit b10bcd350efb55de95792b369996ba266b3612a9
Author: Andrey Kuznetsov 
Date:   2018-05-07T15:59:56Z

IGNITE-6587: Refactored WAL manager critical threads to use GridWorker.

commit 3154112c8bab70d1ff4fe20639648e8a9c7da373
Author: Andrey Kuznetsov 
Date:   2018-05-07T18:55:25Z

IGNITE-6587: Minor improvements.

commit a44fc8064a9ba8f42e4faacde018ac9bc1193f01
Author: Andrey Kuznetsov 
Date:   2018-05-08T10:52:01Z

IGNITE-6587: Refactored CommunicationWorker to use GridWorker.

commit 0efc5354f33666ac705c738d75955258084091de
Author: Andrey Kuznetsov 
Date:   2018-05-08T15:25:56Z

IGNITE-6587: (WIP) switching to timed waits/polls, updating heartbeats.

commit 48c889441a119ef2192c1ff61485c20f80659e82
Author: Andrey Kuznetsov 
Date:   2018-05-14T13:38:47Z

IGNITE-6587: Heartbeat updates in critical WAL manager workers.

commit 69059bda2071f7d6af0cef30cf8d77b160fc56be
Author: Andrey Kuznetsov 
Date:   2018-05-14T15:55:51Z

IGNITE-6587: (WiP) More heartbeat updates in critical workers.

commit 6fb2edf1fef15bb187481f89eddb3710d5daadba
Author: Andrey Kuznetsov 
Date:   2018-05-15T13:09:16Z

IGNITE-6587: Heartbeat updates in checkpointer.

commit 9cd8cba1fdc8c21d999df4c334daaef7e226aa0f
Author: Andrey Kuznetsov 
Date:   2018-05-15T13:23:17Z

IGNITE-6587: More heartbeat updates in critical workers.

commit 5245abc0c72812c4746127b25638a28dac7af11a
Author: Andrey Kuznetsov 
Date:   2018-05-16T20:14:31Z

IGNITE-6587: Extended StripedExecutor.Stripe from GridWorker.

commit 2eb448f6a7ff7a6be423742336480a7b98793c2d
Author: Andrey Kuznetsov 
Date:   2018-05-17T11:37:08Z

IGNITE-6587: Heartbeat updates in StripedExecutor workers.

commit 8304fa92879fe293f112d1c7ba515b5bd30cc271
Author: Andrey Kuznetsov 
Date:   2018-05-17T12:33:23Z

Merge branch 'master' into ignite-6587

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FsyncModeFileWriteAheadLogManager.java

commit 5051aa4d44e78b5362dd0094bc5a57566eea4855
Author: Andrey Kuznetsov 
Date:   2018-05-17T12:36:23Z

IGNITE-6587: Addition to merge commit.

commit 732d2c25013d94d67188f42b2f96c786eeaf6346
Author: Andrey Kuznetsov 
Date:   2018-05-17T12:42:05Z

IGNITE-6587: Updated WAL FileDecompressors to conform uniform poll style.

commit 0c6f7762f1bc709c1444e545ff6475edc3fd6152
Author: Andrey Kuznetsov 
Date:   2018-05-18T09:54:05Z

IGNITE-6587: Added Checkpointer to worker registry.

commit d64e2f8e97e9d8a339695087ceea1443f8f75141
Author: Andrey Kuznetsov 
Date:   2018-05-18T14:43:22Z

IGNITE-6587: Added Stripe workers to worker registry.

commit fd142e80f80e73d420c0a03923ba3a4783dc7f34
Author: Andrey Kuznetsov 
Date:   2018-05-21T11:40:46Z

IGNITE-6587: Optional reference to worker registry in GridNioServer.

commit de98f8782f1728cc7e4a2852fbd0931cb21d3af9
Author: Andrey Kuznetsov 
Date:   2018-05-21T11:42:11Z

IGNITE-6587: Added worker registry to communication SPI GridNioServer.

commit f24f93abe8b8a3833f556b204c9995d28cd9cf44
Author: Andrey Kuznetsov 
Date:   2018-05-21T13:47:11Z

Merge branch 'master' into ignite-6587

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FsyncModeFileWriteAheadLogManager.java

commit e3a7b9aaed79d45b205715b6b281db702e7be95d
Author: Andrey Kuznetsov 
Date:   2018-05-22T12:30:30Z

IGNITE-6587: Recovering after incorrect merge commit.

commit 45c64a1ebaf5dabcfdf64e35a14542c0bddaa4c2
Author: Andrey Kuznetsov 
Date:   2018-05-22T12:56:27Z

IGNITE-6587: Moved WAL timeout constants to proper place.

commit 95f1ce0a9c3002a72fa47c616f928521570d2d40
Author: Andrey Kuznetsov 
Date:   2018-05-22T13:12:36Z

IGNITE-6587: Made WAL FileDecompressors not critical again.

commit 538fa40bf8f4a6585fecc265f36be54510c9a101
Author: 

[GitHub] asfgit closed pull request #11: IGNITE-9617 Different input possibilities for services page

[GitBox]

asfgit closed pull request #11: IGNITE-9617 Different input possibilities for 
services page
URL: https://github.com/apache/ignite-teamcity-bot/pull/11
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java
index 02da6ca..641c9db 100644
--- a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java
+++ b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/TcHelper.java
@@ -180,6 +180,10 @@ private BranchesTracked getTrackedBranches() {
 ) {
 try (IAnalyticsEnabledTeamcity teamcity = server(srvId, prov)) {
 List builds = 
teamcity.getFinishedBuildsIncludeSnDepFailed(buildTypeId, branchForTc);
+
+if (builds.isEmpty())
+return false;
+
 BuildRef build = builds.get(builds.size() - 1);
 String comment;
 
@@ -193,9 +197,8 @@ private BranchesTracked getTrackedBranches() {
 return false;
 }
 
-if ("finished".equals(build.state)) {
+if ("finished".equals(build.state))
 return teamcity.sendJiraComment(ticket, comment);
-}
 
 return false;
 }
diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/ObserverTask.java
 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/ObserverTask.java
index aa3c2f5..fcc1e86 100644
--- 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/ObserverTask.java
+++ 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/observer/ObserverTask.java
@@ -48,7 +48,7 @@
 try (IAnalyticsEnabledTeamcity teamcity = 
helper.server(info.srvId, info.prov)) {
 Build build = teamcity.getBuild(info.build.getId());
 
-if (!build.state.equals("finished"))
+if (!"finished".equals(build.state))
 continue;
 }
 
diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
index 19077f9..e844300 100644
--- 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
+++ 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
@@ -36,6 +36,7 @@
 import org.apache.ignite.ci.web.CtxListener;
 import org.apache.ignite.ci.web.rest.login.ServiceUnauthorizedException;
 import org.apache.ignite.ci.web.model.SimpleResult;
+import org.apache.ignite.internal.util.typedef.F;
 import org.jetbrains.annotations.NotNull;
 import org.jetbrains.annotations.Nullable;
 
@@ -73,7 +74,7 @@ public SimpleResult triggerBuild(
 jiraRes = observeJira(srvId, branchForTc, ticketId, helper, 
teamcity, build, prov);
 }
 
-return new SimpleResult("Tests started." + (!jiraRes.equals("") ? 
"" + jiraRes : ""));
+return new SimpleResult("Tests started." + (!jiraRes.isEmpty() ? 
"" + jiraRes : ""));
 }
 
 @GET
@@ -112,7 +113,7 @@ public SimpleResult commentJiraEx(@QueryParam("serverId") 
@Nullable String srvId
 
 ticketId = getTicketId(pr);
 
-if (ticketId.equals("")) {
+if (ticketId.isEmpty()) {
 jiraRes = "JIRA ticket can't be commented - " +
 "PR title \"" + pr.getTitle() + "\" should starts with 
\"IGNITE-\"." +
 " Please, rename PR according to the" +
@@ -124,10 +125,10 @@ public SimpleResult commentJiraEx(@QueryParam("serverId") 
@Nullable String srvId
 }
 
 if (helper.notifyJira(srvId, prov, suiteId, branchForTc, "ignite-" + 
ticketId))
-return new SimpleResult("JIRA commented." + (!jiraRes.equals("") ? 
jiraRes : ""));
+return new SimpleResult("JIRA commented." + (!jiraRes.isEmpty() ? 
jiraRes : ""));
 else
 // TODO Write catched exceptions to the response.
-return new SimpleResult("JIRA wasn't commented." + 
(!jiraRes.equals("") ? "" + jiraRes : ""));
+return new SimpleResult("JIRA wasn't commented." + 
(!jiraRes.isEmpty() ? "" + jiraRes : ""));
 }
 
 /**
@@ -149,12 +150,12 @@ private String observeJira(
 Build build,
 ICredentialsProv prov
 ) {
-if (ticketId == null || ticketId.equals("")) {
+if (F.isEmpty(ticketId)) {
 PullRequest pr = teamcity.getPullRequest(branchForTc);
 
 ticketId = getTicketId(pr);
 
-if (ticketId.equals("")) {
+if (ticketId.isEmpty()) {
 return "JIRA ticket will not be notified 

Re: Cache scan efficiency

[Dmitriy Pavlov]

As I understood it is not a hack, it is an advanced feature for warming up
the partition. We can build warm-up of the overall cache by calling its
partitions warm-up. Users often ask about this feature and are not
confident with our lazy upload.

Please correct me if I misunderstood the idea.

пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan :

> I would rather fix the scan than hack the scan. Is there any technical
> reason for hacking it now instead of fixing it properly? Can some of the
> experts in this thread provide an estimate of complexity and difference in
> work that would be required for each approach?
>
> D.
>
> On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
> alexey.goncha...@gmail.com>
> wrote:
>
> > I think it would be beneficial for some Ignite users if we added such a
> > partition warmup method to the public API. The method should be
> > well-documented and state that it may invalidate existing page cache. It
> > will be a very effective instrument until we add the proper scan ability
> > that Vladimir was referring to.
> >
> > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov :
> >
> > > Folks,
> > >
> > > Such warming up can be an effective technique for performing
> calculations
> > > which required large cache
> > > data reads, but I think it's the single narrow use case of all over
> > Ignite
> > > store usages. Like all other
> > > powerfull techniques, we should use it wisely. In the general case, I
> > think
> > > we should consider other
> > > techniques mentioned by Vladimir and may create something like `global
> > > statistics of cache data usage`
> > > to choose the best technique in each case.
> > >
> > > For instance, it's not obvious what would take longer: multi-block
> reads
> > or
> > > 50 single-block reads issues
> > > sequentially. It strongly depends on used hardware under the hood and
> > might
> > > depend on workload system
> > > resources (CPU-intensive calculations and I\O access) as well. But
> > > `statistics` will help us to choose
> > > the right way.
> > >
> > >
> > > On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov 
> > wrote:
> > >
> > > > Hi Alexei,
> > > >
> > > > I did not find any PRs associated with the ticket for check code
> > changes
> > > > behind this idea. Are there any PRs?
> > > >
> > > > If we create some forwards scan of pages, it should be a very
> > > intellectual
> > > > algorithm including a lot of parameters (how much RAM is free, how
> > > probably
> > > > we will need next page, etc). We had the private talk about such idea
> > > some
> > > > time ago.
> > > >
> > > > By my experience, Linux systems already do such forward reading of
> file
> > > > data (for corresponding sequential flagged file descriptors), but
> some
> > > > prefetching of data at the level of application may be useful for
> > > O_DIRECT
> > > > file descriptors.
> > > >
> > > > And one more concern from me is about selecting a right place in the
> > > system
> > > > to do such prefetch.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > вс, 16 сент. 2018 г. в 19:54, Vladimir Ozerov  >:
> > > >
> > > > > HI Alex,
> > > > >
> > > > > This is good that you observed speedup. But I do not think this
> > > solution
> > > > > works for the product in general case. Amount of RAM is limited,
> and
> > > > even a
> > > > > single partition may need more space than RAM available. Moving a
> lot
> > > of
> > > > > pages to page memory for scan means that you evict a lot of other
> > > pages,
> > > > > what will ultimately lead to bad performance of subsequent queries
> > and
> > > > > defeat LRU algorithms, which are of great improtance for good
> > database
> > > > > performance.
> > > > >
> > > > > Database vendors choose another approach - skip BTrees, iterate
> > > direclty
> > > > > over data pages, read them in multi-block fashion, use separate
> scan
> > > > buffer
> > > > > to avoid excessive evictions of other hot pages. Corresponding
> ticket
> > > for
> > > > > SQL exists [1], but idea is common for all parts of the system,
> > > requiring
> > > > > scans.
> > > > >
> > > > > As far as proposed solution, it might be good idea to add special
> API
> > > to
> > > > > "warmup" partition with clear explanation of pros (fast scan after
> > > > warmup)
> > > > > and cons (slowdown of any other operations). But I think we should
> > not
> > > > make
> > > > > this approach part of normal scans.
> > > > >
> > > > > Vladimir.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-6057
> > > > >
> > > > >
> > > > > On Sun, Sep 16, 2018 at 6:44 PM Alexei Scherbakov <
> > > > > alexey.scherbak...@gmail.com> wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > My use case involves scenario where it's necessary to iterate
> over
> > > > > > large(many TBs) persistent cache doing some calculation on read
> > data.
> > > > > >
> > > > > > The basic solution is to iterate cache using ScanQuery.
> > > > > >
> > > > > > This turns out to be slow 

[GitHub] ignite pull request #4774: Ignite 8619 experimental

[oignatenko]

GitHub user oignatenko opened a pull request:

https://github.com/apache/ignite/pull/4774

Ignite 8619 experimental

Temporary PR to try out particular test change at Teamcity

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
ignite-8619-experimental

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4774.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4774


commit 1eeca908a8076a8317947dac8a46845964d1d7ea
Author: Oleg Ignatenko 
Date:   2018-08-23T13:13:28Z

IGNITE-9348 ML examples improvements
- wip (logging improved)
-- verified with diffs overview, executing the examples and studying 
execution logs

commit e50b89c392568ba9b93935c4fa6c7f7f93f5ec6f
Author: Oleg Ignatenko 
Date:   2018-08-23T14:45:57Z

Revert "IGNITE-9348 ML examples improvements"

This reverts commit 1eeca908a8076a8317947dac8a46845964d1d7ea.

commit 474024b4c5bbdb3c0a4ed2f0a66238c8054c6674
Author: Oleg Ignatenko 
Date:   2018-08-23T14:57:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9642b233b5701bdad47ebea163079160227c582a
Author: Oleg Ignatenko 
Date:   2018-08-28T14:01:11Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 7fc16d013ab725d2ff2e1a1b042c983f11d0c4d4
Author: Oleg Ignatenko 
Date:   2018-08-28T15:13:02Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit d2caba67b156674f051f50faebeafe0871bf0914
Author: Oleg Ignatenko 
Date:   2018-08-29T13:14:07Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 16775dff51d71ea68b4a3dea98be552130c493ed
Author: Oleg Ignatenko 
Date:   2018-08-30T09:00:56Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aedb59929974fe205b949225c1a338c68c60cfc8
Author: Oleg Ignatenko 
Date:   2018-08-30T09:42:38Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 39c6482fcdca506aa33011ed21c98060b4a8c68b
Author: Oleg Ignatenko 
Date:   2018-08-30T11:28:05Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 33b32a2760a6559c78283b927e3191180d8ed9e1
Author: Oleg Ignatenko 
Date:   2018-08-30T12:31:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9531028ddd1aef9e95f7e8c8b528086739bbb1b0
Author: Oleg Ignatenko 
Date:   2018-08-30T14:06:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 28f22c6e2fffcb82717ba5da7be2cfd39715c4e3
Author: Oleg Ignatenko 
Date:   2018-08-30T16:41:51Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aacac88db519187413b0fc5ff9d0e55b8f8cff22
Author: Oleg Ignatenko 
Date:   2018-08-31T10:12:32Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 897f920dde46022849b13f9fb86dba8e54119a56
Author: Oleg Ignatenko 
Date:   2018-08-31T13:57:14Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 114c79e54c1b316006ccc3ff22d20d902f9313df
Author: Oleg Ignatenko 
Date:   2018-08-31T17:39:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fd6b659bb8be1992618ce4ce91f568a0988b3afa
Author: Oleg Ignatenko 
Date:   2018-09-02T06:11:42Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 6ae6d1d3cf9743d8d466be0330511ddc8589e944
Author: Oleg Ignatenko 
Date:   2018-09-03T10:27:35Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit e8b27dbd3d0c1cbdb7a7659175f5c7bb447482bf
Author: Oleg Ignatenko 
Date:   2018-09-04T09:49:44Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 622c82efdd0aa182fadea6b7ffa5d4615521a3f5
Author: Oleg Ignatenko 
Date:   2018-09-05T10:50:28Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fb844fe3751e2e8ae87e6b8030b0e4bd659df9c2
Author: Oleg Ignatenko 
Date:   2018-09-05T11:45:58Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 480ed78869277d7e32f725ab71bec9621f1ac03a
Author: Oleg Ignatenko 
Date:   2018-09-06T07:52:55Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit c99762498f617c0e98ea3062a43c0b30092166ef
Author: Oleg Ignatenko 
Date:   2018-09-06T14:45:04Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 2e17175225c72f747d370b5fee96f8be69d6d2cb
Author: Oleg Ignatenko 
Date:   2018-09-06T17:33:54Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9ebcd9a2fe5966b0bf42a95484395867c15d863f
Author: Oleg Ignatenko 
Date:   2018-09-07T13:38:51Z

Merge 

[jira] [Created] (IGNITE-9622) MVCC Cache API: prohibit non PESSIMISTIC REPEATABLE_READ transactions

[Ivan Pavlukhin (JIRA)]

Ivan Pavlukhin created IGNITE-9622:
--

 Summary: MVCC Cache API: prohibit non PESSIMISTIC REPEATABLE_READ 
transactions
 Key: IGNITE-9622
 URL: https://issues.apache.org/jira/browse/IGNITE-9622
 Project: Ignite
  Issue Type: Bug
  Components: mvcc
Reporter: Ivan Pavlukhin


For cache with enabled mvcc we should allow running only PESSIMISTIC 
REPEATABLE_READ transactions. For other modes exception should be thrown (like 
is done for SQL API).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [MTCGA]: new failures in builds [1888723] needs to be handled

[Dmitriy Pavlov]

Hi Roman Kondakov, Vladimir Ozerov,

is it possible and is it reasonable to fix tests faster than full support
in https://issues.apache.org/jira/browse/IGNITE-9390 ?

E.g. can the Parity test be fixed using
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Tests+How+To#IgniteTestsHowTo-Testof.NETAPIparitywithJavaAPI
 recommendations?

Sincerely,
Dmitriy Pavlov

пн, 17 сент. 2018 г. в 17:25, :

> Hi Ignite Developer,
>
> I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed.
> I hope you can help.
>
>  *New test failure in master
> IgniteConfigurationParityTest.TestIgniteConfiguration
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7406999233661835096=%3Cdefault%3E=testDetails
>
>  *New test failure in master DataRegionMetricsTest.TestMemoryMetrics
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6742613397597284603=%3Cdefault%3E=testDetails
>
>  *New test failure in master MemoryMetricsTest.TestMemoryMetrics
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7558087625238261420=%3Cdefault%3E=testDetails
>  Changes may led to failure were done by
>  - kondakov87
> http://ci.ignite.apache.org/viewModification.html?modId=831703=false
>
> - If your changes can led to this failure(s), please create issue
> with label MakeTeamCityGreenAgain and assign it to you.
> -- If you have fix, please set ticket to PA state and write to dev
> list fix is ready
> -- For case fix will require some time please mute test and set
> label Muted_Test to issue
> - If you know which change caused failure please contact change
> author directly
> - If you don't know which change caused failure please send
> message to dev list to find out
> Should you have any questions please contact dev@ignite.apache.org
> Best Regards,
> MTCGA.Bot
> Notification generated at Mon Sep 17 17:25:16 MSK 2018
>

Re: [MTCGA]: new failures in builds [1871897] needs to be handled

[Dmitrii Ryabov]

Looks like problem I had described in the ticket.

https://issues.apache.org/jira/browse/IGNITE-7618?focusedCommentId=16506923=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16506923

2018-09-15 12:01 GMT+03:00 Dmitriy Pavlov :

> Dmitriy G, Ivan B,
>
> could you please double-check if failure is not coming from
> https://issues.apache.org/jira/browse/IGNITE-7618
>
> Sincerely,
> Dmitriy Pavlov
>
> сб, 15 сент. 2018 г. в 5:42, :
>
> > Hi Ignite Developer,
> >
> > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed.
> > I hope you can help.
> >
> >  *New test failure in master IgniteStandByClusterTest.testSimple
> > https://ci.ignite.apache.org/project.html?projectId=
> IgniteTests24Java8=1332314705000986815=%
> 3Cdefault%3E=testDetails
> >  Changes may led to failure were done by
> >  - bessonov.ip
> > http://ci.ignite.apache.org/viewModification.html?modId=
> 831651=false
> >
> > - If your changes can led to this failure(s), please create issue
> > with label MakeTeamCityGreenAgain and assign it to you.
> > -- If you have fix, please set ticket to PA state and write to
> dev
> > list fix is ready
> > -- For case fix will require some time please mute test and set
> > label Muted_Test to issue
> > - If you know which change caused failure please contact change
> > author directly
> > - If you don't know which change caused failure please send
> > message to dev list to find out
> > Should you have any questions please contact dev@ignite.apache.org
> > Best Regards,
> > MTCGA.Bot
> > Notification generated at Sat Sep 15 05:42:21 MSK 2018
> >
>

Re: Cache scan efficiency

[Dmitriy Setrakyan]

I would rather fix the scan than hack the scan. Is there any technical
reason for hacking it now instead of fixing it properly? Can some of the
experts in this thread provide an estimate of complexity and difference in
work that would be required for each approach?

D.

On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk 
wrote:

> I think it would be beneficial for some Ignite users if we added such a
> partition warmup method to the public API. The method should be
> well-documented and state that it may invalidate existing page cache. It
> will be a very effective instrument until we add the proper scan ability
> that Vladimir was referring to.
>
> пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov :
>
> > Folks,
> >
> > Such warming up can be an effective technique for performing calculations
> > which required large cache
> > data reads, but I think it's the single narrow use case of all over
> Ignite
> > store usages. Like all other
> > powerfull techniques, we should use it wisely. In the general case, I
> think
> > we should consider other
> > techniques mentioned by Vladimir and may create something like `global
> > statistics of cache data usage`
> > to choose the best technique in each case.
> >
> > For instance, it's not obvious what would take longer: multi-block reads
> or
> > 50 single-block reads issues
> > sequentially. It strongly depends on used hardware under the hood and
> might
> > depend on workload system
> > resources (CPU-intensive calculations and I\O access) as well. But
> > `statistics` will help us to choose
> > the right way.
> >
> >
> > On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov 
> wrote:
> >
> > > Hi Alexei,
> > >
> > > I did not find any PRs associated with the ticket for check code
> changes
> > > behind this idea. Are there any PRs?
> > >
> > > If we create some forwards scan of pages, it should be a very
> > intellectual
> > > algorithm including a lot of parameters (how much RAM is free, how
> > probably
> > > we will need next page, etc). We had the private talk about such idea
> > some
> > > time ago.
> > >
> > > By my experience, Linux systems already do such forward reading of file
> > > data (for corresponding sequential flagged file descriptors), but some
> > > prefetching of data at the level of application may be useful for
> > O_DIRECT
> > > file descriptors.
> > >
> > > And one more concern from me is about selecting a right place in the
> > system
> > > to do such prefetch.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > вс, 16 сент. 2018 г. в 19:54, Vladimir Ozerov :
> > >
> > > > HI Alex,
> > > >
> > > > This is good that you observed speedup. But I do not think this
> > solution
> > > > works for the product in general case. Amount of RAM is limited, and
> > > even a
> > > > single partition may need more space than RAM available. Moving a lot
> > of
> > > > pages to page memory for scan means that you evict a lot of other
> > pages,
> > > > what will ultimately lead to bad performance of subsequent queries
> and
> > > > defeat LRU algorithms, which are of great improtance for good
> database
> > > > performance.
> > > >
> > > > Database vendors choose another approach - skip BTrees, iterate
> > direclty
> > > > over data pages, read them in multi-block fashion, use separate scan
> > > buffer
> > > > to avoid excessive evictions of other hot pages. Corresponding ticket
> > for
> > > > SQL exists [1], but idea is common for all parts of the system,
> > requiring
> > > > scans.
> > > >
> > > > As far as proposed solution, it might be good idea to add special API
> > to
> > > > "warmup" partition with clear explanation of pros (fast scan after
> > > warmup)
> > > > and cons (slowdown of any other operations). But I think we should
> not
> > > make
> > > > this approach part of normal scans.
> > > >
> > > > Vladimir.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-6057
> > > >
> > > >
> > > > On Sun, Sep 16, 2018 at 6:44 PM Alexei Scherbakov <
> > > > alexey.scherbak...@gmail.com> wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > My use case involves scenario where it's necessary to iterate over
> > > > > large(many TBs) persistent cache doing some calculation on read
> data.
> > > > >
> > > > > The basic solution is to iterate cache using ScanQuery.
> > > > >
> > > > > This turns out to be slow because iteration over cache involves a
> lot
> > > of
> > > > > random disk access for reading data pages referenced from leaf
> pages
> > by
> > > > > links.
> > > > >
> > > > > This is especially true when data is stored on disks with slow
> random
> > > > > access, like SAS disks. In my case on modern SAS disks array
> reading
> > > > speed
> > > > > was like several MB/sec while sequential read speed in perf test
> was
> > > > about
> > > > > GB/sec.
> > > > >
> > > > > I was able to fix the issue by using ScanQuery with explicit
> > partition
> > > > set
> > > > > and running simple warmup code before each partition scan.
> > > > >
> > > 

Re: loadcache very very slow

[Alexey Kuznetsov]

Indeed.

You should ask such questions on user list.
Just start a thread there + attach a DDL script of your schema in SQL
Server.

-- 
Alexey Kuznetsov

[jira] [Created] (IGNITE-9621) MVCC: sqlline warning that transactions are not supported

[Stepan Pilschikov (JIRA)]

Stepan Pilschikov created IGNITE-9621:
-

 Summary: MVCC: sqlline warning that transactions are not supported
 Key: IGNITE-9621
 URL: https://issues.apache.org/jira/browse/IGNITE-9621
 Project: Ignite
  Issue Type: Bug
  Components: mvcc
Affects Versions: 2.7
Reporter: Stepan Pilschikov


MVCC enabled
--autoCommit=true

In sqlline first initial lines throwing warning message:
```
issuing: !connect jdbc:ignite:thin://127.0.0.1:10800 '' '' 
org.apache.ignite.IgniteJdbcThinDriver
Connecting to jdbc:ignite:thin://127.0.0.1:10800
Connected to: Apache Ignite (version 2.7.1#19700101-sha1:)
Driver: Apache Ignite Thin JDBC Driver (version 
2.7.1#19700101-sha1:)
Autocommit status: false
Sep 17, 2018 5:44:32 PM org.apache.ignite.internal.jdbc.thin.JdbcThinConnection 
setTransactionIsolation
WARNING: Transactions are not supported.
Sep 17, 2018 5:44:32 PM org.apache.ignite.internal.jdbc.thin.JdbcThinConnection 
getTransactionIsolation
WARNING: Transactions are not supported.
Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline version 1.3.0
0: jdbc:ignite:thin://127.0.0.1:10800>
```



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4766: PR

[akuznetsov-gridgain]

Github user akuznetsov-gridgain closed the pull request at:

https://github.com/apache/ignite/pull/4766


---

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218104124
 
 

 ##
 File path: ignite-tc-helper-web/src/main/webapp/statistics.html
 ##
 @@ -0,0 +1,213 @@
+
+
+Ignite Teamcity - statistics master's branch
+
+https://code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css;>
+
+
+https://code.jquery.com/jquery-1.12.4.js";>
+https://code.jquery.com/ui/1.12.1/jquery-ui.js";>
+
+
+
+table {
+width: 70%;
+border-collapse: collapse;
+}
+td, th {
+
+padding: 10px 5px 10px 5px;
+}
+th {
+text-align: left;
+padding: 5px;
+background-color: #f5f5ff;
+color: #00;
+}
+tr:nth-child(odd) { background-color: #fafaff; }
+
+
+

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218102484
 
 

 ##
 File path: ignite-tc-helper-web/src/main/webapp/statistics.html
 ##
 @@ -0,0 +1,213 @@
+
+
+Ignite Teamcity - statistics master's branch
+
+https://code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css;>
+
+
+https://code.jquery.com/jquery-1.12.4.js";>
+https://code.jquery.com/ui/1.12.1/jquery-ui.js";>
+
+
+

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218102782
 
 

 ##
 File path: ignite-tc-helper-web/src/main/webapp/statistics.html
 ##
 @@ -0,0 +1,213 @@
+
+
+Ignite Teamcity - statistics master's branch
+
+https://code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css;>
+
+
+https://code.jquery.com/jquery-1.12.4.js";>
+https://code.jquery.com/ui/1.12.1/jquery-ui.js";>
+
+
+
+table {
+width: 70%;
+border-collapse: collapse;
+}
+td, th {
+
+padding: 10px 5px 10px 5px;
+}
+th {
+text-align: left;
+padding: 5px;
+background-color: #f5f5ff;
+color: #00;
+}
+tr:nth-child(odd) { background-color: #fafaff; }
+
+
+

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218099007
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcmodel/result/issues/IssueRef.java
 ##
 @@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.tcmodel.result.issues;
+
+import javax.xml.bind.annotation.XmlAttribute;
+
+/**
+ * Issue short version from list of build's related issues.
+ *
+ * See example of XML, e.g. here
+ * https://ci.ignite.apache.org/app/rest/latest/builds/id:1694977/relatedIssues
+ */
+public class IssueRef {
+@XmlAttribute public String id;
+@XmlAttribute public String url;
+
+@Override public String toString() {
 
 Review comment:
   Missed javadoc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (IGNITE-9620) MVCC: select throwing `Transaction is already completed` exception after mvcc missmatch

[Stepan Pilschikov (JIRA)]

Stepan Pilschikov created IGNITE-9620:
-

 Summary: MVCC: select  throwing `Transaction is already completed` 
exception after mvcc missmatch
 Key: IGNITE-9620
 URL: https://issues.apache.org/jira/browse/IGNITE-9620
 Project: Ignite
  Issue Type: Bug
  Components: mvcc
Affects Versions: 2.7
Reporter: Stepan Pilschikov


Using sqlline with autoCommit=false

{code}
switch to first user
- select * from test:
result: [[1, 1, test_1]]
switch to second user
- insert into test(id, field_int, field_var) values (2, 2, 'test_2'):
- commit:
- select * from test:
result: [[1, 1, test_1], [2, 2, test_2]]
switch to first user
- insert into test(id, field_int, field_var) values (2, 2, 'test_2'):
error: Mvcc version mismatch.
- select * from test
{code}

During last select throwing exception
{code}
0: jdbc:ignite:thin://127.0.0.1:10800> select * from test;
select * from test;
Error: Transaction is already completed. (state=25000,code=0)
java.sql.SQLException: Transaction is already completed.
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:764)
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:212)
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:475)
at sqlline.Commands.execute(Commands.java:823)
at sqlline.Commands.sql(Commands.java:733)
at sqlline.SqlLine.dispatch(SqlLine.java:795)
at sqlline.SqlLine.begin(SqlLine.java:668)
at sqlline.SqlLine.start(SqlLine.java:373)
at sqlline.SqlLine.main(SqlLine.java:265)
{code}

Exception in node logs:
{code}
[17:44:36,234][SEVERE][jdbc-request-handler-worker-#61][JdbcRequestHandler] 
Failed to execute SQL query [reqId=0, req=JdbcQueryExecuteRequest 
[schemaName=PUBLIC, pageSize=1024, maxRows=0, sqlQry=select * from test, 
args=Object[] [], stmtType=ANY_STATEMENT_TYPE, autoCommit=false]]
class org.apache.ignite.internal.processors.query.IgniteSQLException: 
Transaction is already completed.
at 
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.checkActive(MvccUtils.java:623)
at 
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.requestSnapshot(MvccUtils.java:780)
at 
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.mvccTracker(MvccUtils.java:761)
at 
org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.mvccTracker(MvccUtils.java:744)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.runQueryTwoStep(IgniteH2Indexing.java:1731)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.doRunDistributedQuery(IgniteH2Indexing.java:2521)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:2074)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2139)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2134)
at 
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2711)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2148)
at 
org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.executeQuery(JdbcRequestHandler.java:511)
at 
org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.doHandle(JdbcRequestHandler.java:245)
at 
org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandlerWorker.body(JdbcRequestHandlerWorker.java:90)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
{code}

Works for any query which are throwing mvcc missmatch exception
After commit select query works again



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218099233
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcmodel/result/issues/IssuesUsagesList.java
 ##
 @@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.tcmodel.result.issues;
+
+import java.util.Collections;
+import java.util.List;
+import javax.xml.bind.annotation.XmlElement;
+import javax.xml.bind.annotation.XmlRootElement;
+
+/**
+ * List of build's related issues from TC.
+ *
+ * See example of XML, e.g. here
+ * https://ci.ignite.apache.org/app/rest/latest/builds/id:1694977/relatedIssues
+ */
+@XmlRootElement(name = "issuesUsages")
+public class IssuesUsagesList {
+@XmlElement(name = "issueUsage")
+private List issuesUsages;
+
+@XmlElement Integer count;
+
+@XmlElement String href;
+
+public List getIssuesUsagesNonNull() {
+return issuesUsages == null ? Collections.emptyList() : issuesUsages;
+}
+
+@Override public String toString() {
 
 Review comment:
   Missed javadoc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218100191
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/model/current/BuildStatisticsSummary.java
 ##
 @@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.web.model.current;
+
+import java.text.DateFormat;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import javax.annotation.Nonnull;
+import org.apache.ignite.ci.ITeamcity;
+import org.apache.ignite.ci.tcmodel.hist.BuildRef;
+import org.apache.ignite.ci.tcmodel.result.Build;
+import org.apache.ignite.ci.tcmodel.result.TestOccurrencesRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueUsage;
+import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrence;
+import org.apache.ignite.ci.util.TimeUtil;
+import org.apache.ignite.ci.web.IBackgroundUpdatable;
+
+/**
+ * Summary of build statistics.
+ */
+public class BuildStatisticsSummary extends UpdateInfo implements 
IBackgroundUpdatable {
+/** Short problem names. */
+public static final String TOTAL = "TOTAL";
+
+private static Map shortProblemNames = new HashMap<>();
+
+static {
+shortProblemNames.put(TOTAL, "TT");
+shortProblemNames.put(ProblemOccurrence.TC_EXECUTION_TIMEOUT, "ET");
+shortProblemNames.put(ProblemOccurrence.TC_JVM_CRASH, "JC");
+shortProblemNames.put(ProblemOccurrence.TC_OOME, "OO");
+shortProblemNames.put(ProblemOccurrence.TC_EXIT_CODE, "EC");
+shortProblemNames.put(ProblemOccurrence.TC_FAILED_TESTS, "FT");
+shortProblemNames.put(ProblemOccurrence.SNAPSHOT_DEPENDENCY_ERROR, 
"SD");
+shortProblemNames.put(ProblemOccurrence.OTHER, "OT");
+}
+
+/** Build with test and problems references. */
+public Integer buildId;
+
+public String date;
+
+/** Test occurrences. */
+public TestOccurrencesRef testOccurrences;
+
+/** List of problem occurrences. */
+private List problemOccurrenceList;
+
+/** Snapshot-dependencies build run result. */
+public Map> dependenciesProblems;
+
+/** List of related issues. */
+private List relatedIssues;
+
+/** Duration printable. */
+public String durationPrintable;
+
+/** Short build run result (without snapshot-dependencies printable 
result). */
+public Map totalProblems;
+
+/** Build problems count. */
+public long problemsCount;
+
+/** Is fake stub. */
+public boolean isFakeStub;
+
+/**
+ * @param buildId Build id.
+ */
+public BuildStatisticsSummary(Integer buildId){
+this.buildId = buildId;
+}
+
+/** Initialize build statistics. */
+public void initialize(@Nonnull final ITeamcity teamcity) {
+Build build = teamcity.getBuild(buildId);
+
+DateFormat dateFormat = new SimpleDateFormat("dd-MM-'T'HH:mm:ss");
+dateFormat.format(build.getFinishDate());
+date = dateFormat.format(build.getFinishDate());
+
+isFakeStub = build.isFakeStub();
+
+if (isFakeStub)
+return;
+
+testOccurrences = build.testOccurrences;
+
+relatedIssues = 
teamcity.getIssuesUsagesList(build.relatedIssuesRef.href).getIssuesUsagesNonNull().stream()
+.map(IssueUsage::getIssue).collect(Collectors.toList());
+
+durationPrintable = TimeUtil
+.getDurationPrintable(build.getFinishDate().getTime() - 
build.getStartDate().getTime());
+
+List snapshotDependencies = 
getSnapshotDependencies(teamcity, build);
+
+List snapshotDependenciesWithProblems = 
getBuildsWithProblems(snapshotDependencies);
+
+problemOccurrenceList = getProblems(teamcity, 
snapshotDependenciesWithProblems);
+
+

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218100964
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/build/GetBuildTestFailures.java
 ##
 @@ -149,4 +153,71 @@ public TestFailuresSummary getBuildTestFails(
 
 return res;
 }
+
+@GET
+@Path("history")
+public List getBuildsHistory(
+@Nullable @QueryParam("server") String server,
+@Nullable @QueryParam("buildType") String buildType,
+@Nullable @QueryParam("branch") String branch,
+@Nullable @QueryParam("sinceDate") String sinceDate,
+@Nullable @QueryParam("untilDate") String untilDate)
+throws ServiceUnauthorizedException, ParseException {
+
+DateFormat dateFormat = new SimpleDateFormat("ddMMHHmmss");
+
+String srvId = isNullOrEmpty(server) ? "apache" : server;
+String buildTypeId = isNullOrEmpty(buildType) ? 
"IgniteTests24Java8_RunAll" : buildType;
+String branchName = isNullOrEmpty(branch) ? "refs/heads/master" : 
branch;
+Date sinceDateFilter = isNullOrEmpty(sinceDate) ? null : 
dateFormat.parse(sinceDate);
+Date untilDateFilter = isNullOrEmpty(untilDate) ? null : 
dateFormat.parse(untilDate);
+
+final BackgroundUpdater updater = 
CtxListener.getBackgroundUpdater(context);
+
+final ITcHelper tcHelper = CtxListener.getTcHelper(context);
+
+final ICredentialsProv prov = ICredentialsProv.get(req);
+
+try (IAnalyticsEnabledTeamcity teamcity = tcHelper.server(srvId, 
prov)) {
+
+int[] finishedBuilds = 
teamcity.getBuildNumbersFromHistory(buildTypeId, branchName, sinceDateFilter, 
untilDateFilter);
+
+List buildsStatistics = new ArrayList<>();
+
+for (int i = 0; i < finishedBuilds.length; i++) {
+int buildId = finishedBuilds[i];
+
+FullQueryParams param = new FullQueryParams();
+param.setBuildId(buildId);
+param.setBranch(branchName);
+param.setServerId(srvId);
+
+BuildStatisticsSummary buildsStatistic = updater.get(
+BUILDS_STATISTICS_SUMMARY_CACHE_NAME, prov, param,
+(k) -> getBuildStatisticsSummaryNoCache(srvId, buildId), 
false);
+
+if (!buildsStatistic.isFakeStub)
+buildsStatistics.add(buildsStatistic);
+}
+
+return buildsStatistics;
+}
+}
+
+private BuildStatisticsSummary getBuildStatisticsSummaryNoCache(String 
server, int buildId) {
+
 
 Review comment:
   Redundant blank line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218096032
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
 ##
 @@ -591,6 +625,24 @@ private void addTestOccurrencesToStat(TestOccurrences 
val, String normalizedBran
 });
 }
 
+@Override public IssuesUsagesList getIssuesUsagesList(String href) {
 
 Review comment:
   Missec javadoc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218091720
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
 ##
 @@ -197,7 +203,7 @@ private IgnitePersistentTeamcity(Ignite ignite, 
IgniteTeamcityHelper teamcity) {
 }
 
 /**
- * @return Build history: {@link BuildRef} lists cache, 32 parts, 
transaactional
+ * @return Build history: {@link BuildRef} lists cache, 32 parts, 
transactional
 
 Review comment:
   Missed dot at the end of sentence.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval

[GitBox]

SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218104050
 
 

 ##
 File path: ignite-tc-helper-web/src/main/webapp/statistics.html
 ##
 @@ -0,0 +1,213 @@
+
+
+Ignite Teamcity - statistics master's branch
+
+https://code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css;>
+
+
+https://code.jquery.com/jquery-1.12.4.js";>
+https://code.jquery.com/ui/1.12.1/jquery-ui.js";>
+
+
+
+table {
+width: 70%;
+border-collapse: collapse;
+}
+td, th {
+
+padding: 10px 5px 10px 5px;
+}
+th {
+text-align: left;
+padding: 5px;
+background-color: #f5f5ff;
+color: #00;
+}
+tr:nth-child(odd) { background-color: #fafaff; }
+
+
+
+
+var g_shownDataHashCodeHex = ""
+var g_updTimer = null;
+
+$(document).ready(function() {
+$.getScript("js/testfails-2.0.js", function(data, textStatus, jqxhr){ 
});
+
+$( document ).tooltip();
+loadData();
+//todo fix setInterval( function() { checkForUpdate(); }, 3);
+
+$.ajax({ url: "rest/branches/version",  success: showVersionInfo, 
error: showErrInLoadStatus });
+
+if(g_updTimer==null) {
+g_updTimer=setTimeout(tstTimeout, 3200);
+}
+setInterval(tstTimeout, 1);
+});
+
+function tstTimeout() {
+
+if(g_updTimer!=null) {
+clearTimeout(g_updTimer);
+g_updTimer=null;
+}
+
+
+var d = new Date();
+var n = d.getTime();
+
+// $(document.body).prepend("timeout at "+n+"
");
+
+
+if(g_updTimer==null) {
+g_updTimer=setTimeout(tstTimeout, 3200);
+}
+}
+
+function parmsForRest() {
+var curReqParms = "";
+var server = findGetParameter("server");
+if(server!=null) {
+curReqParms += "&server=" + server;
+}
+
+var buildType = findGetParameter("buildType");
+if(buildType!=null) {
+curReqParms += "&buildType=" + buildType;
+}
+
+var branch = findGetParameter("branch");
+if(branch!=null) {
+curReqParms += "&branch=" + branch;
+}
+
+var count = findGetParameter("count");
+if(count!=null) {
+curReqParms += "&count=" + count;
+}
+
+var sinceDate = findGetParameter("sinceDate");
+if(sinceDate!=null) {
+curReqParms += "&sinceDate=" + sinceDate;
+}
+
+var untilDate = findGetParameter("untilDate");
+if(untilDate!=null) {
+curReqParms += "&untilDate=" + untilDate;
+}
+
+curReqParms = curReqParms.replace("&","?");
+
+return curReqParms;
+}
+
+function loadData() {
+var curFailuresUrl = "rest/build/history" + parmsForRest();
+
+$("#loadStatus").html(" Please wait");
+$.ajax({
+url: curFailuresUrl,
+success: function(result) {
+if(result.updateRequired || 
(isDefinedAndFilled(result.runningUpdates) && result.runningUpdates>0)) {
+setTimeout(checkForUpdate, 3000)
+$("#loadStatus").html(" Updating");
+} else {
+$("#loadStatus").html("");
+}
+showData(result);
+g_shownDataHashCodeHex = 
isDefinedAndFilled(result.hashCodeHex) ? result.hashCodeHex : "";
+},
+error: showErrInLoadStatus
+});
+}
+function showData(result) {
+$("#statistics").html(showBuildHistory(result));
+}
+
+function median(arr){
+arr = arr.sort(function(a, b){ return a - b; });
+var i = arr.length / 2;
+return i % 1 == 0 ? (arr[i - 1] + arr[i]) / 2 : arr[Math.floor(i)];
+}
+
+function formatString(string) {
+if (string.indexOf("_") < string.indexOf(":"))
+string = string.substring(string.indexOf("_") + 1);
+
+return "" + string.substring(0, string.indexOf(":") + 1) + "" + 
string.substring(string.indexOf(":") + 1);
+}
+
+function showBuildHistory(result) {
+var res = "" +
+"";
+
+
[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218102850
 
 

 ##
 File path: ignite-tc-helper-web/src/main/webapp/statistics.html
 ##
 @@ -0,0 +1,213 @@
+
+
+Ignite Teamcity - statistics master's branch
+
+https://code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css;>
+
+
+https://code.jquery.com/jquery-1.12.4.js";>
+https://code.jquery.com/ui/1.12.1/jquery-ui.js";>
+
+
+
+table {
+width: 70%;
+border-collapse: collapse;
+}
+td, th {
+
+padding: 10px 5px 10px 5px;
+}
+th {
+text-align: left;
+padding: 5px;
+background-color: #f5f5ff;
+color: #00;
+}
+tr:nth-child(odd) { background-color: #fafaff; }
+
+
+


[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218099174
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcmodel/result/issues/IssueUsage.java
 ##
 @@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.tcmodel.result.issues;
+
+import javax.xml.bind.annotation.XmlAccessType;
+import javax.xml.bind.annotation.XmlAccessorType;
+import javax.xml.bind.annotation.XmlElement;
+import javax.xml.bind.annotation.XmlRootElement;
+import org.apache.ignite.ci.tcmodel.changes.ChangesList;
+
+/**
+ * Build's related issue from TC.
+ *
+ * See example of XML, e.g. here
+ * https://ci.ignite.apache.org/app/rest/latest/builds/id:1694977/relatedIssues
+ */
+@XmlRootElement(name = "IssueUsage")
+@XmlAccessorType(XmlAccessType.FIELD)
+public class IssueUsage {
+@XmlElement(name = "issue")
+private IssueRef issue;
+@XmlElement(name = "changes")
+private ChangesList changesList;
+
+public IssueRef getIssue(){
 
 Review comment:
   Missed javadoc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218102209
 
 

 ##
 File path: ignite-tc-helper-web/src/main/webapp/css/style-1.5.css
 ##
 @@ -115,4 +115,33 @@ form li:after
clear:both;
display:block;
margin-top:10px;
+}
+.stat{
 
 Review comment:
   Missed blank lines between styles.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218100491
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/model/current/BuildStatisticsSummary.java
 ##
 @@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.web.model.current;
+
+import java.text.DateFormat;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import javax.annotation.Nonnull;
+import org.apache.ignite.ci.ITeamcity;
+import org.apache.ignite.ci.tcmodel.hist.BuildRef;
+import org.apache.ignite.ci.tcmodel.result.Build;
+import org.apache.ignite.ci.tcmodel.result.TestOccurrencesRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueUsage;
+import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrence;
+import org.apache.ignite.ci.util.TimeUtil;
+import org.apache.ignite.ci.web.IBackgroundUpdatable;
+
+/**
+ * Summary of build statistics.
+ */
+public class BuildStatisticsSummary extends UpdateInfo implements 
IBackgroundUpdatable {
+/** Short problem names. */
+public static final String TOTAL = "TOTAL";
+
+private static Map shortProblemNames = new HashMap<>();
+
+static {
+shortProblemNames.put(TOTAL, "TT");
+shortProblemNames.put(ProblemOccurrence.TC_EXECUTION_TIMEOUT, "ET");
+shortProblemNames.put(ProblemOccurrence.TC_JVM_CRASH, "JC");
+shortProblemNames.put(ProblemOccurrence.TC_OOME, "OO");
+shortProblemNames.put(ProblemOccurrence.TC_EXIT_CODE, "EC");
+shortProblemNames.put(ProblemOccurrence.TC_FAILED_TESTS, "FT");
+shortProblemNames.put(ProblemOccurrence.SNAPSHOT_DEPENDENCY_ERROR, 
"SD");
+shortProblemNames.put(ProblemOccurrence.OTHER, "OT");
+}
+
+/** Build with test and problems references. */
+public Integer buildId;
+
+public String date;
+
+/** Test occurrences. */
+public TestOccurrencesRef testOccurrences;
+
+/** List of problem occurrences. */
+private List problemOccurrenceList;
+
+/** Snapshot-dependencies build run result. */
+public Map> dependenciesProblems;
+
+/** List of related issues. */
+private List relatedIssues;
+
+/** Duration printable. */
+public String durationPrintable;
+
+/** Short build run result (without snapshot-dependencies printable 
result). */
+public Map totalProblems;
+
+/** Build problems count. */
+public long problemsCount;
+
+/** Is fake stub. */
+public boolean isFakeStub;
+
+/**
+ * @param buildId Build id.
+ */
+public BuildStatisticsSummary(Integer buildId){
+this.buildId = buildId;
+}
+
+/** Initialize build statistics. */
+public void initialize(@Nonnull final ITeamcity teamcity) {
+Build build = teamcity.getBuild(buildId);
+
+DateFormat dateFormat = new SimpleDateFormat("dd-MM-'T'HH:mm:ss");
+dateFormat.format(build.getFinishDate());
+date = dateFormat.format(build.getFinishDate());
+
+isFakeStub = build.isFakeStub();
+
+if (isFakeStub)
+return;
+
+testOccurrences = build.testOccurrences;
+
+relatedIssues = 
teamcity.getIssuesUsagesList(build.relatedIssuesRef.href).getIssuesUsagesNonNull().stream()
+.map(IssueUsage::getIssue).collect(Collectors.toList());
+
+durationPrintable = TimeUtil
+.getDurationPrintable(build.getFinishDate().getTime() - 
build.getStartDate().getTime());
+
+List snapshotDependencies = 
getSnapshotDependencies(teamcity, build);
+
+List snapshotDependenciesWithProblems = 
getBuildsWithProblems(snapshotDependencies);
+
+problemOccurrenceList = getProblems(teamcity, 
snapshotDependenciesWithProblems);
+
+

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218100329
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/model/current/BuildStatisticsSummary.java
 ##
 @@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.web.model.current;
+
+import java.text.DateFormat;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import javax.annotation.Nonnull;
+import org.apache.ignite.ci.ITeamcity;
+import org.apache.ignite.ci.tcmodel.hist.BuildRef;
+import org.apache.ignite.ci.tcmodel.result.Build;
+import org.apache.ignite.ci.tcmodel.result.TestOccurrencesRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueUsage;
+import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrence;
+import org.apache.ignite.ci.util.TimeUtil;
+import org.apache.ignite.ci.web.IBackgroundUpdatable;
+
+/**
+ * Summary of build statistics.
+ */
+public class BuildStatisticsSummary extends UpdateInfo implements 
IBackgroundUpdatable {
+/** Short problem names. */
+public static final String TOTAL = "TOTAL";
+
+private static Map shortProblemNames = new HashMap<>();
+
+static {
+shortProblemNames.put(TOTAL, "TT");
+shortProblemNames.put(ProblemOccurrence.TC_EXECUTION_TIMEOUT, "ET");
+shortProblemNames.put(ProblemOccurrence.TC_JVM_CRASH, "JC");
+shortProblemNames.put(ProblemOccurrence.TC_OOME, "OO");
+shortProblemNames.put(ProblemOccurrence.TC_EXIT_CODE, "EC");
+shortProblemNames.put(ProblemOccurrence.TC_FAILED_TESTS, "FT");
+shortProblemNames.put(ProblemOccurrence.SNAPSHOT_DEPENDENCY_ERROR, 
"SD");
+shortProblemNames.put(ProblemOccurrence.OTHER, "OT");
+}
+
+/** Build with test and problems references. */
+public Integer buildId;
+
+public String date;
+
+/** Test occurrences. */
+public TestOccurrencesRef testOccurrences;
+
+/** List of problem occurrences. */
+private List problemOccurrenceList;
+
+/** Snapshot-dependencies build run result. */
+public Map> dependenciesProblems;
+
+/** List of related issues. */
+private List relatedIssues;
+
+/** Duration printable. */
+public String durationPrintable;
+
+/** Short build run result (without snapshot-dependencies printable 
result). */
+public Map totalProblems;
+
+/** Build problems count. */
+public long problemsCount;
+
+/** Is fake stub. */
+public boolean isFakeStub;
+
+/**
+ * @param buildId Build id.
+ */
+public BuildStatisticsSummary(Integer buildId){
+this.buildId = buildId;
+}
+
+/** Initialize build statistics. */
+public void initialize(@Nonnull final ITeamcity teamcity) {
+Build build = teamcity.getBuild(buildId);
+
+DateFormat dateFormat = new SimpleDateFormat("dd-MM-'T'HH:mm:ss");
+dateFormat.format(build.getFinishDate());
+date = dateFormat.format(build.getFinishDate());
+
+isFakeStub = build.isFakeStub();
+
+if (isFakeStub)
+return;
+
+testOccurrences = build.testOccurrences;
+
+relatedIssues = 
teamcity.getIssuesUsagesList(build.relatedIssuesRef.href).getIssuesUsagesNonNull().stream()
+.map(IssueUsage::getIssue).collect(Collectors.toList());
+
+durationPrintable = TimeUtil
+.getDurationPrintable(build.getFinishDate().getTime() - 
build.getStartDate().getTime());
+
+List snapshotDependencies = 
getSnapshotDependencies(teamcity, build);
+
+List snapshotDependenciesWithProblems = 
getBuildsWithProblems(snapshotDependencies);
+
+problemOccurrenceList = getProblems(teamcity, 
snapshotDependenciesWithProblems);
+
+

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218100654
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/model/current/BuildStatisticsSummary.java
 ##
 @@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.web.model.current;
+
+import java.text.DateFormat;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import javax.annotation.Nonnull;
+import org.apache.ignite.ci.ITeamcity;
+import org.apache.ignite.ci.tcmodel.hist.BuildRef;
+import org.apache.ignite.ci.tcmodel.result.Build;
+import org.apache.ignite.ci.tcmodel.result.TestOccurrencesRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueUsage;
+import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrence;
+import org.apache.ignite.ci.util.TimeUtil;
+import org.apache.ignite.ci.web.IBackgroundUpdatable;
+
+/**
+ * Summary of build statistics.
+ */
+public class BuildStatisticsSummary extends UpdateInfo implements 
IBackgroundUpdatable {
+/** Short problem names. */
+public static final String TOTAL = "TOTAL";
+
+private static Map shortProblemNames = new HashMap<>();
+
+static {
+shortProblemNames.put(TOTAL, "TT");
+shortProblemNames.put(ProblemOccurrence.TC_EXECUTION_TIMEOUT, "ET");
+shortProblemNames.put(ProblemOccurrence.TC_JVM_CRASH, "JC");
+shortProblemNames.put(ProblemOccurrence.TC_OOME, "OO");
+shortProblemNames.put(ProblemOccurrence.TC_EXIT_CODE, "EC");
+shortProblemNames.put(ProblemOccurrence.TC_FAILED_TESTS, "FT");
+shortProblemNames.put(ProblemOccurrence.SNAPSHOT_DEPENDENCY_ERROR, 
"SD");
+shortProblemNames.put(ProblemOccurrence.OTHER, "OT");
+}
+
+/** Build with test and problems references. */
+public Integer buildId;
+
+public String date;
+
+/** Test occurrences. */
+public TestOccurrencesRef testOccurrences;
+
+/** List of problem occurrences. */
+private List problemOccurrenceList;
+
+/** Snapshot-dependencies build run result. */
+public Map> dependenciesProblems;
+
+/** List of related issues. */
+private List relatedIssues;
+
+/** Duration printable. */
+public String durationPrintable;
+
+/** Short build run result (without snapshot-dependencies printable 
result). */
+public Map totalProblems;
+
+/** Build problems count. */
+public long problemsCount;
+
+/** Is fake stub. */
+public boolean isFakeStub;
+
+/**
+ * @param buildId Build id.
+ */
+public BuildStatisticsSummary(Integer buildId){
+this.buildId = buildId;
+}
+
+/** Initialize build statistics. */
+public void initialize(@Nonnull final ITeamcity teamcity) {
+Build build = teamcity.getBuild(buildId);
+
+DateFormat dateFormat = new SimpleDateFormat("dd-MM-'T'HH:mm:ss");
+dateFormat.format(build.getFinishDate());
+date = dateFormat.format(build.getFinishDate());
+
+isFakeStub = build.isFakeStub();
+
+if (isFakeStub)
+return;
+
+testOccurrences = build.testOccurrences;
+
+relatedIssues = 
teamcity.getIssuesUsagesList(build.relatedIssuesRef.href).getIssuesUsagesNonNull().stream()
+.map(IssueUsage::getIssue).collect(Collectors.toList());
+
+durationPrintable = TimeUtil
+.getDurationPrintable(build.getFinishDate().getTime() - 
build.getStartDate().getTime());
+
+List snapshotDependencies = 
getSnapshotDependencies(teamcity, build);
+
+List snapshotDependenciesWithProblems = 
getBuildsWithProblems(snapshotDependencies);
+
+problemOccurrenceList = getProblems(teamcity, 
snapshotDependenciesWithProblems);
+
+

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218103132
 
 

 ##
 File path: ignite-tc-helper-web/src/main/webapp/statistics.html
 ##
 @@ -0,0 +1,213 @@
+
+
+Ignite Teamcity - statistics master's branch
+
+https://code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css;>
+
+
+https://code.jquery.com/jquery-1.12.4.js";>
+https://code.jquery.com/ui/1.12.1/jquery-ui.js";>
+
+
+
+table {
+width: 70%;
+border-collapse: collapse;
+}
+td, th {
+
+padding: 10px 5px 10px 5px;
+}
+th {
+text-align: left;
+padding: 5px;
+background-color: #f5f5ff;
+color: #00;
+}
+tr:nth-child(odd) { background-color: #fafaff; }
+
+
+


[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218100744
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/model/current/BuildStatisticsSummary.java
 ##
 @@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.web.model.current;
+
+import java.text.DateFormat;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import javax.annotation.Nonnull;
+import org.apache.ignite.ci.ITeamcity;
+import org.apache.ignite.ci.tcmodel.hist.BuildRef;
+import org.apache.ignite.ci.tcmodel.result.Build;
+import org.apache.ignite.ci.tcmodel.result.TestOccurrencesRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueUsage;
+import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrence;
+import org.apache.ignite.ci.util.TimeUtil;
+import org.apache.ignite.ci.web.IBackgroundUpdatable;
+
+/**
+ * Summary of build statistics.
+ */
+public class BuildStatisticsSummary extends UpdateInfo implements 
IBackgroundUpdatable {
+/** Short problem names. */
+public static final String TOTAL = "TOTAL";
+
+private static Map shortProblemNames = new HashMap<>();
+
+static {
+shortProblemNames.put(TOTAL, "TT");
+shortProblemNames.put(ProblemOccurrence.TC_EXECUTION_TIMEOUT, "ET");
+shortProblemNames.put(ProblemOccurrence.TC_JVM_CRASH, "JC");
+shortProblemNames.put(ProblemOccurrence.TC_OOME, "OO");
+shortProblemNames.put(ProblemOccurrence.TC_EXIT_CODE, "EC");
+shortProblemNames.put(ProblemOccurrence.TC_FAILED_TESTS, "FT");
+shortProblemNames.put(ProblemOccurrence.SNAPSHOT_DEPENDENCY_ERROR, 
"SD");
+shortProblemNames.put(ProblemOccurrence.OTHER, "OT");
+}
+
+/** Build with test and problems references. */
+public Integer buildId;
+
+public String date;
+
+/** Test occurrences. */
+public TestOccurrencesRef testOccurrences;
+
+/** List of problem occurrences. */
+private List problemOccurrenceList;
+
+/** Snapshot-dependencies build run result. */
+public Map> dependenciesProblems;
+
+/** List of related issues. */
+private List relatedIssues;
+
+/** Duration printable. */
+public String durationPrintable;
+
+/** Short build run result (without snapshot-dependencies printable 
result). */
+public Map totalProblems;
+
+/** Build problems count. */
+public long problemsCount;
+
+/** Is fake stub. */
+public boolean isFakeStub;
+
+/**
+ * @param buildId Build id.
+ */
+public BuildStatisticsSummary(Integer buildId){
+this.buildId = buildId;
+}
+
+/** Initialize build statistics. */
+public void initialize(@Nonnull final ITeamcity teamcity) {
+Build build = teamcity.getBuild(buildId);
+
+DateFormat dateFormat = new SimpleDateFormat("dd-MM-'T'HH:mm:ss");
+dateFormat.format(build.getFinishDate());
+date = dateFormat.format(build.getFinishDate());
+
+isFakeStub = build.isFakeStub();
+
+if (isFakeStub)
+return;
+
+testOccurrences = build.testOccurrences;
+
+relatedIssues = 
teamcity.getIssuesUsagesList(build.relatedIssuesRef.href).getIssuesUsagesNonNull().stream()
+.map(IssueUsage::getIssue).collect(Collectors.toList());
+
+durationPrintable = TimeUtil
+.getDurationPrintable(build.getFinishDate().getTime() - 
build.getStartDate().getTime());
+
+List snapshotDependencies = 
getSnapshotDependencies(teamcity, build);
+
+List snapshotDependenciesWithProblems = 
getBuildsWithProblems(snapshotDependencies);
+
+problemOccurrenceList = getProblems(teamcity, 
snapshotDependenciesWithProblems);
+
+

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218095484
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
 ##
 @@ -290,6 +300,26 @@ private IgnitePersistentTeamcity(Ignite ignite, 
IgniteTeamcityHelper teamcity) {
 
 return mergeByIdToHistoricalOrder(persistedValue, builds);
 });
+
+if (sinceDate != null && untilDate != null)
 
 Review comment:
   Is it possible to get request with `sinceDate` and without `untilDate`? And 
vice versa?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218090754
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
 ##
 @@ -62,7 +65,19 @@
  * @param branch
  * @return list of builds in historical order, recent builds coming last
  */
-List getFinishedBuilds(String projectId, String branch);
+default List getFinishedBuilds(String projectId, String branch) {
+return getFinishedBuilds(projectId, branch, null, null);
+};
+
+/**
+ * @param projectId suite ID (string without spaces)
+ * @param branch
 
 Review comment:
   Missed param description.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218098809
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcmodel/result/Build.java
 ##
 @@ -127,4 +137,4 @@ public Triggered getTriggered() {
 public void setTriggered(Triggered triggered) {
 this.triggered = triggered;
 }
-}
+}
 
 Review comment:
   File should ends with a blank line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218096565
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgniteTeamcityHelper.java
 ##
 @@ -354,18 +357,39 @@ public String basicAuthToken() {
 private List getBuildHistory(@Nullable String buildTypeId,
 @Nullable String branchName,
 boolean dfltFilter,
-@Nullable String state) {
+@Nullable String state){
+
+return getBuildHistory(buildTypeId, branchName, dfltFilter, state, 
null, null);
+}
+
+private List getBuildHistory(@Nullable String buildTypeId,
+@Nullable String branchName,
+boolean dfltFilter,
+@Nullable String state,
+@Nullable Date sinceDate,
+@Nullable Date untilDate) {
 String btFilter = isNullOrEmpty(buildTypeId) ? "" : ",buildType:" + 
buildTypeId + "";
 String stateFilter = isNullOrEmpty(state) ? "" : (",state:" + state);
-String brachFilter = isNullOrEmpty(branchName) ? "" :",branch:" + 
branchName;
+String branchFilter = isNullOrEmpty(branchName) ? "" :",branch:" + 
branchName;
+String sinceDateFilter = sinceDate == null ? "" : ",sinceDate:" + 
getDateYyyyMmDdTHhMmSsZ(sinceDate);
+String untilDateFilter = untilDate == null ? "" : ",untilDate:" + 
getDateYyyyMmDdTHhMmSsZ(untilDate);
 
 return sendGetXmlParseJaxb(host + "app/rest/latest/builds"
 + "?locator="
 + "defaultFilter:" + dfltFilter
 + btFilter
 + stateFilter
-+ brachFilter
-+ ",count:1000", Builds.class).getBuildsNonNull();
++ branchFilter
++ ",count:" + DEFAULT_BUILDS_COUNT
++ sinceDateFilter
++ untilDateFilter, Builds.class).getBuildsNonNull();
+}
+
+public String getDateYyyyMmDdTHhMmSsZ(Date date){
+
 
 Review comment:
   Redundant blank line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218098330
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcmodel/hist/BuildRef.java
 ##
 @@ -52,6 +52,8 @@
 
 @XmlAttribute public Boolean composite;
 
+@XmlAttribute public String webUrl;
 
 Review comment:
   Rebase your branch on current master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218099774
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/model/current/BuildStatisticsSummary.java
 ##
 @@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.web.model.current;
+
+import java.text.DateFormat;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import javax.annotation.Nonnull;
+import org.apache.ignite.ci.ITeamcity;
+import org.apache.ignite.ci.tcmodel.hist.BuildRef;
+import org.apache.ignite.ci.tcmodel.result.Build;
+import org.apache.ignite.ci.tcmodel.result.TestOccurrencesRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueUsage;
+import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrence;
+import org.apache.ignite.ci.util.TimeUtil;
+import org.apache.ignite.ci.web.IBackgroundUpdatable;
+
+/**
+ * Summary of build statistics.
+ */
+public class BuildStatisticsSummary extends UpdateInfo implements 
IBackgroundUpdatable {
+/** Short problem names. */
+public static final String TOTAL = "TOTAL";
+
+private static Map shortProblemNames = new HashMap<>();
+
+static {
+shortProblemNames.put(TOTAL, "TT");
+shortProblemNames.put(ProblemOccurrence.TC_EXECUTION_TIMEOUT, "ET");
+shortProblemNames.put(ProblemOccurrence.TC_JVM_CRASH, "JC");
+shortProblemNames.put(ProblemOccurrence.TC_OOME, "OO");
+shortProblemNames.put(ProblemOccurrence.TC_EXIT_CODE, "EC");
+shortProblemNames.put(ProblemOccurrence.TC_FAILED_TESTS, "FT");
+shortProblemNames.put(ProblemOccurrence.SNAPSHOT_DEPENDENCY_ERROR, 
"SD");
+shortProblemNames.put(ProblemOccurrence.OTHER, "OT");
+}
+
+/** Build with test and problems references. */
+public Integer buildId;
+
+public String date;
+
+/** Test occurrences. */
+public TestOccurrencesRef testOccurrences;
+
+/** List of problem occurrences. */
+private List problemOccurrenceList;
+
+/** Snapshot-dependencies build run result. */
+public Map> dependenciesProblems;
+
+/** List of related issues. */
+private List relatedIssues;
+
+/** Duration printable. */
+public String durationPrintable;
+
+/** Short build run result (without snapshot-dependencies printable 
result). */
+public Map totalProblems;
+
+/** Build problems count. */
+public long problemsCount;
+
+/** Is fake stub. */
+public boolean isFakeStub;
+
+/**
+ * @param buildId Build id.
+ */
+public BuildStatisticsSummary(Integer buildId){
+this.buildId = buildId;
+}
+
+/** Initialize build statistics. */
+public void initialize(@Nonnull final ITeamcity teamcity) {
+Build build = teamcity.getBuild(buildId);
+
+DateFormat dateFormat = new SimpleDateFormat("dd-MM-'T'HH:mm:ss");
+dateFormat.format(build.getFinishDate());
+date = dateFormat.format(build.getFinishDate());
+
+isFakeStub = build.isFakeStub();
+
+if (isFakeStub)
+return;
+
+testOccurrences = build.testOccurrences;
+
+relatedIssues = 
teamcity.getIssuesUsagesList(build.relatedIssuesRef.href).getIssuesUsagesNonNull().stream()
+.map(IssueUsage::getIssue).collect(Collectors.toList());
+
+durationPrintable = TimeUtil
+.getDurationPrintable(build.getFinishDate().getTime() - 
build.getStartDate().getTime());
+
+List snapshotDependencies = 
getSnapshotDependencies(teamcity, build);
+
+List snapshotDependenciesWithProblems = 
getBuildsWithProblems(snapshotDependencies);
+
+problemOccurrenceList = getProblems(teamcity, 
snapshotDependenciesWithProblems);
+
+

[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218097013
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgniteTeamcityHelper.java
 ##
 @@ -409,12 +444,12 @@ public ChangesList getChangesList(String href) {
 }
 
 /** {@inheritDoc} */
-@Override public List getFinishedBuilds(String projectId,
-String branch) {
+public List getFinishedBuilds(String projectId,
+String branch, Date sinceDate, Date untilDate) {
 List finished = getBuildHistory(projectId,
 UrlUtil.escape(branch),
 true,
-null);
+null, sinceDate, untilDate);
 
 Review comment:
   Move args to the new lines.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218097631
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgniteTeamcityHelper.java
 ##
 @@ -409,12 +444,12 @@ public ChangesList getChangesList(String href) {
 }
 
 /** {@inheritDoc} */
-@Override public List getFinishedBuilds(String projectId,
-String branch) {
+public List getFinishedBuilds(String projectId,
+String branch, Date sinceDate, Date untilDate) {
 
 Review comment:
   Place all args on the same line or different lines.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218096888
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgniteTeamcityHelper.java
 ##
 @@ -378,7 +402,16 @@ public Build getBuild(String href) {
 }
 
 public ProblemOccurrences getProblems(Build build) {
-return getJaxbUsingHref(build.problemOccurrences.href, 
ProblemOccurrences.class);
+if (build.problemOccurrences != null) {
+ProblemOccurrences problemOccurrences = 
getJaxbUsingHref(build.problemOccurrences.href, ProblemOccurrences.class);
+
+problemOccurrences.problemOccurrences
+.forEach(p -> p.buildRef = build);
 
 Review comment:
   No need new line here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218091080
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
 ##
 @@ -80,7 +97,11 @@
 CompletableFuture> getQueuedBuilds(@Nullable String branch);
 
 default int[] getBuildNumbersFromHistory(String projectId, String 
branchNameForHist) {
-return getFinishedBuilds(projectId, 
branchNameForHist).stream().mapToInt(BuildRef::getId).toArray();
+return getBuildNumbersFromHistory(projectId, branchNameForHist, null, 
null);
+}
+
+default int[] getBuildNumbersFromHistory(String projectId, String 
branchNameForHist, Date sinceDate, Date untilDate) {
 
 Review comment:
   Missec javadoc.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218091389
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
 ##
 @@ -52,6 +54,7 @@
 public interface ITeamcity extends AutoCloseable {
 
 String DEFAULT = "";
+long DEFAULT_BUILDS_COUNT = 1000;
 
 Review comment:
   Missec javadoc. Need empty line before previous field.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[GitHub] SomeFire commented on a change in pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval


2018-09-17

Thread
GitBox
SomeFire commented on a change in pull request #9: IGNITE-9541 Add the 
comparison for two general statistics "RunAll" for master in the date interval
URL: https://github.com/apache/ignite-teamcity-bot/pull/9#discussion_r218090321
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
 ##
 @@ -62,7 +65,19 @@
  * @param branch
  * @return list of builds in historical order, recent builds coming last
  */
-List getFinishedBuilds(String projectId, String branch);
+default List getFinishedBuilds(String projectId, String branch) {
+return getFinishedBuilds(projectId, branch, null, null);
+};
+
+/**
+ * @param projectId suite ID (string without spaces)
+ * @param branch
+ * @param sinceDate
+ * @param untilDate
+ * @return list of builds in historical order, recent builds coming last
+ */
+
 
 Review comment:
   Remove empty line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



Re: IgniteDataStreamer - into table and not key\value


2018-09-17

Thread
Ilya Kasnacheev
Hello!

You don't need to remove those properties, just remove package names:



Please try and read mails carefully.

Regards,
-- 
Ilya Kasnacheev


пн, 17 сент. 2018 г. в 17:05, wt :

> this is making no sense. There is absolutely nothing in the documentation
> that says remove those items from the configuration. All i am looking for
> is
> an example of a table that can be loaded with the data streamer without the
> data ending up outside of the table. I have tried with and without that
> simple mapper in th config and it makes no difference.
>
> simple example i have:
>
> 1) config (has no caches specified - contents placed at the end)
> 2) create table in jdbc
>
> CREATE TABLE IF NOT EXISTS Testy (
>   id int,
>   name varchar,
>PRIMARY KEY (id)
> ) WITH "template=partitioned,backups=0";
>
> 3) table is visible in ignite
>
> {
> "successStatus": 0,
> "error": null,
> "sessionToken": null,
> "response": [
> {
> "cacheName": "SQL_PUBLIC_TESTY",
> "types": [
>
> "SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a"
> ],
> "keyClasses": {
>
> "SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a":
> "java.lang.Integer"
> },
> "valClasses": {
>
> "SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a":
> "java.lang.Object"
> },
> "fields": {
>
> "SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a": {
> "ID": "java.lang.Integer",
> "NAME": "java.lang.String"
> }
> },
> "indexes": {
>
> "SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a": []
> }
> }}}
>
>
> 4) simple .net project
>
>
> public class Testy
> {
>
> [QuerySqlField]
> public int id { get; set; }
> [QuerySqlField]
> public string name { get; set; }
>
> }
>
> var cfg = new IgniteConfiguration()
> {
> SpringConfigUrl = @"C:\Cluster-client.xml"
> };
>
> var ignite = Ignition.Start(cfg);
> using (var ldr = ignite.GetDataStreamer Testy>("SQL_PUBLIC_TESTY"))
> {
> ldr.AddData(1, new Testy() {id=1, name = "" });
> }
>
>
> 5) db beaver
>
> SELECT count(*) FROM testy (result is 0)
>
> 6) rest query
>
>
> http://127.0.0.1:8080/ignite?cmd=get=SQL_PUBLIC_TESTY=int=1
>
> {
> "successStatus": 0,
> "affinityNodeId": "9a006702-e11e-4fdc-bf84-7f0b28fc498f",
> "error": null,
> "sessionToken": null,
> "response": {
> "name": "",
> "id": 1
> }
> }
>
>
>
>
> -
> config file
>
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xmlns:util="http://www.springframework.org/schema/util;
>xsi:schemaLocation="http://www.springframework.org/schema/beans
>
> http://www.springframework.org/schema/beans/spring-beans.xsd
>http://www.springframework.org/schema/util
>
> http://www.springframework.org/schema/util/spring-util.xsd;>
>
> 
> 
> 
>
> 
>  class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> 
> 
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
> 
> 
> 127.0.0.1:47500..47510
> 
> 
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.configuration.BinaryConfiguration">
> 
>  class="org.apache.ignite.binary.BinaryBasicNameMapper">
> 
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
> 
>  class="org.apache.ignite.configuration.DataRegionConfiguration">
> 
> 
> 
> 
> 
> 
>
> 
> 
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>



[GitHub] ignite pull request #4773: IGNITE-9492 Ignore SingleMessage updated where th...


2018-09-17

Thread
Jokser
GitHub user Jokser opened a pull request:

https://github.com/apache/ignite/pull/4773

IGNITE-9492 Ignore SingleMessage updated where there is in-progress exchange

…rogress.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9492

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4773.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4773


commit 0d0a25364eb4dfa97f57e98fa56e1e996476c604
Author: Pavel Kovalenko 
Date:   2018-09-17T14:46:45Z

IGNITE-9492 Ignore SingleMessage updated where there is exchange in 
progress.




---



[jira] [Created] (IGNITE-9619) Document REST "getall" array format


2018-09-17

Thread
Ilya Kasnacheev (JIRA)
Ilya Kasnacheev created IGNITE-9619:
---

 Summary: Document REST "getall" array format
 Key: IGNITE-9619
 URL: https://issues.apache.org/jira/browse/IGNITE-9619
 Project: Ignite
  Issue Type: Improvement
  Components: documentation
Reporter: Ilya Kasnacheev
 Fix For: 2.7


https://apacheignite.readme.io/docs/rest-api#get-all <-- this page should have 
a section about new IGNITE_REST_GETALL_AS_ARRAY System Property, as well as an 
example of response after it is set.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



Re: [MTCGA]: new failures in builds [1888722] needs to be handled


2018-09-17

Thread
Dmitriy Pavlov
Hi Igniters,

Roma Kondakov in private talk confirmed that test is failed because of
recent changes.

To be fixed under ticket: https://issues.apache.org/jira/browse/IGNITE-9390

Sincerely,
Dmitriy Pavlov

пн, 17 сент. 2018 г. в 16:42, :

> Hi Ignite Developer,
>
> I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed.
> I hope you can help.
>
>  *New test failure in master
> IgniteConfigurationParityTest.TestIgniteConfiguration
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2475516713290452471=%3Cdefault%3E=testDetails
>
>  *New test failure in master DataRegionMetricsTest.TestMemoryMetrics
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-808679864195727798=%3Cdefault%3E=testDetails
>  Changes may led to failure were done by
>  - kondakov87
> http://ci.ignite.apache.org/viewModification.html?modId=831703=false
>
> - If your changes can led to this failure(s), please create issue
> with label MakeTeamCityGreenAgain and assign it to you.
> -- If you have fix, please set ticket to PA state and write to dev
> list fix is ready
> -- For case fix will require some time please mute test and set
> label Muted_Test to issue
> - If you know which change caused failure please contact change
> author directly
> - If you don't know which change caused failure please send
> message to dev list to find out
> Should you have any questions please contact dev@ignite.apache.org
> Best Regards,
> MTCGA.Bot
> Notification generated at Mon Sep 17 16:41:57 MSK 2018
>



[MTCGA]:  new failures in builds [1888723] needs to be handled


2018-09-17

Thread
dpavlov . tasks
Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New test failure in master 
IgniteConfigurationParityTest.TestIgniteConfiguration 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7406999233661835096=%3Cdefault%3E=testDetails

 *New test failure in master DataRegionMetricsTest.TestMemoryMetrics 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6742613397597284603=%3Cdefault%3E=testDetails

 *New test failure in master MemoryMetricsTest.TestMemoryMetrics 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7558087625238261420=%3Cdefault%3E=testDetails
 Changes may led to failure were done by 
 - kondakov87 
http://ci.ignite.apache.org/viewModification.html?modId=831703=false

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Mon Sep 17 17:25:16 MSK 2018 



Re: IgniteDataStreamer - into table and not key\value


2018-09-17

Thread
wt
this is making no sense. There is absolutely nothing in the documentation
that says remove those items from the configuration. All i am looking for is
an example of a table that can be loaded with the data streamer without the
data ending up outside of the table. I have tried with and without that
simple mapper in th config and it makes no difference.

simple example i have:

1) config (has no caches specified - contents placed at the end)
2) create table in jdbc 

CREATE TABLE IF NOT EXISTS Testy (
  id int,
  name varchar,
   PRIMARY KEY (id)
) WITH "template=partitioned,backups=0";

3) table is visible in ignite

{
"successStatus": 0,
"error": null,
"sessionToken": null,
"response": [
{
"cacheName": "SQL_PUBLIC_TESTY",
"types": [

"SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a"
],
"keyClasses": {

"SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a":
"java.lang.Integer"
},
"valClasses": {

"SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a":
"java.lang.Object"
},
"fields": {

"SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a": {
"ID": "java.lang.Integer",
"NAME": "java.lang.String"
}
},
"indexes": {

"SQL_PUBLIC_TESTY_07052f5a_37ac_4992_8a19_79da93e3df8a": []
}
}}}


4) simple .net project


public class Testy
{

[QuerySqlField]
public int id { get; set; }
[QuerySqlField]
public string name { get; set; }

}

var cfg = new IgniteConfiguration()
{
SpringConfigUrl = @"C:\Cluster-client.xml"
};

var ignite = Ignition.Start(cfg);
using (var ldr = ignite.GetDataStreamer("SQL_PUBLIC_TESTY"))
{
ldr.AddData(1, new Testy() {id=1, name = "" });
}


5) db beaver

SELECT count(*) FROM testy (result is 0)

6) rest query

http://127.0.0.1:8080/ignite?cmd=get=SQL_PUBLIC_TESTY=int=1

{
"successStatus": 0,
"affinityNodeId": "9a006702-e11e-4fdc-bf84-7f0b28fc498f",
"error": null,
"sessionToken": null,
"response": {
"name": "",
"id": 1
}
}




-
config file

http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xmlns:util="http://www.springframework.org/schema/util;
   xsi:schemaLocation="http://www.springframework.org/schema/beans
  
http://www.springframework.org/schema/beans/spring-beans.xsd
   http://www.springframework.org/schema/util
  
http://www.springframework.org/schema/util/spring-util.xsd;>











127.0.0.1:47500..47510



































--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/



[jira] [Created] (IGNITE-9618) Need to be replace the data compression algorithm


2018-09-17

Thread
Alexand Polyakov (JIRA)
Alexand Polyakov created IGNITE-9618:


 Summary: Need to be replace the data compression algorithm
 Key: IGNITE-9618
 URL: https://issues.apache.org/jira/browse/IGNITE-9618
 Project: Ignite
  Issue Type: New Feature
  Components: persistence
Reporter: Alexand Polyakov


Now used zip and its speed slow
Exist alternatives and on tests in terms of performance they showed themselves 
to be better
source file wal 1Gb
result
||algoritm||time, ms||size, byte||
|zip|18 889|79 950 283|
|[Snappy|https://github.com/xerial/snappy-java]|3 372|156 482 623|
|[lz4|https://github.com/lz4/lz4-java]|2 047|128 591 795|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



[GitHub] ignite pull request #4772: Refine CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT ...


2018-09-17

Thread
pavlukhin
GitHub user pavlukhin opened a pull request:

https://github.com/apache/ignite/pull/4772

Refine CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT javadoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
mvcc-transactional-snapshot-doc-refinement

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4772.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4772


commit a6ba2870a581ace6b7f21511348136fb1fdd9d18
Author: ipavlukhin 
Date:   2018-09-17T13:52:41Z

correct CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT javadoc




---



[MTCGA]:  new failures in builds [1888722] needs to be handled


2018-09-17

Thread
dpavlov . tasks
Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I 
hope you can help.

 *New test failure in master 
IgniteConfigurationParityTest.TestIgniteConfiguration 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2475516713290452471=%3Cdefault%3E=testDetails

 *New test failure in master DataRegionMetricsTest.TestMemoryMetrics 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-808679864195727798=%3Cdefault%3E=testDetails
 Changes may led to failure were done by 
 - kondakov87 
http://ci.ignite.apache.org/viewModification.html?modId=831703=false

- If your changes can led to this failure(s), please create issue with 
label MakeTeamCityGreenAgain and assign it to you.
-- If you have fix, please set ticket to PA state and write to dev list 
fix is ready 
-- For case fix will require some time please mute test and set label 
Muted_Test to issue 
- If you know which change caused failure please contact change author 
directly
- If you don't know which change caused failure please send message to 
dev list to find out
Should you have any questions please contact dev@ignite.apache.org 
Best Regards,
MTCGA.Bot 
Notification generated at Mon Sep 17 16:41:57 MSK 2018 



Re: Cache scan efficiency


2018-09-17

Thread
Alexey Goncharuk
I think it would be beneficial for some Ignite users if we added such a
partition warmup method to the public API. The method should be
well-documented and state that it may invalidate existing page cache. It
will be a very effective instrument until we add the proper scan ability
that Vladimir was referring to.

пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov :

> Folks,
>
> Such warming up can be an effective technique for performing calculations
> which required large cache
> data reads, but I think it's the single narrow use case of all over Ignite
> store usages. Like all other
> powerfull techniques, we should use it wisely. In the general case, I think
> we should consider other
> techniques mentioned by Vladimir and may create something like `global
> statistics of cache data usage`
> to choose the best technique in each case.
>
> For instance, it's not obvious what would take longer: multi-block reads or
> 50 single-block reads issues
> sequentially. It strongly depends on used hardware under the hood and might
> depend on workload system
> resources (CPU-intensive calculations and I\O access) as well. But
> `statistics` will help us to choose
> the right way.
>
>
> On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov  wrote:
>
> > Hi Alexei,
> >
> > I did not find any PRs associated with the ticket for check code changes
> > behind this idea. Are there any PRs?
> >
> > If we create some forwards scan of pages, it should be a very
> intellectual
> > algorithm including a lot of parameters (how much RAM is free, how
> probably
> > we will need next page, etc). We had the private talk about such idea
> some
> > time ago.
> >
> > By my experience, Linux systems already do such forward reading of file
> > data (for corresponding sequential flagged file descriptors), but some
> > prefetching of data at the level of application may be useful for
> O_DIRECT
> > file descriptors.
> >
> > And one more concern from me is about selecting a right place in the
> system
> > to do such prefetch.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вс, 16 сент. 2018 г. в 19:54, Vladimir Ozerov :
> >
> > > HI Alex,
> > >
> > > This is good that you observed speedup. But I do not think this
> solution
> > > works for the product in general case. Amount of RAM is limited, and
> > even a
> > > single partition may need more space than RAM available. Moving a lot
> of
> > > pages to page memory for scan means that you evict a lot of other
> pages,
> > > what will ultimately lead to bad performance of subsequent queries and
> > > defeat LRU algorithms, which are of great improtance for good database
> > > performance.
> > >
> > > Database vendors choose another approach - skip BTrees, iterate
> direclty
> > > over data pages, read them in multi-block fashion, use separate scan
> > buffer
> > > to avoid excessive evictions of other hot pages. Corresponding ticket
> for
> > > SQL exists [1], but idea is common for all parts of the system,
> requiring
> > > scans.
> > >
> > > As far as proposed solution, it might be good idea to add special API
> to
> > > "warmup" partition with clear explanation of pros (fast scan after
> > warmup)
> > > and cons (slowdown of any other operations). But I think we should not
> > make
> > > this approach part of normal scans.
> > >
> > > Vladimir.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-6057
> > >
> > >
> > > On Sun, Sep 16, 2018 at 6:44 PM Alexei Scherbakov <
> > > alexey.scherbak...@gmail.com> wrote:
> > >
> > > > Igniters,
> > > >
> > > > My use case involves scenario where it's necessary to iterate over
> > > > large(many TBs) persistent cache doing some calculation on read data.
> > > >
> > > > The basic solution is to iterate cache using ScanQuery.
> > > >
> > > > This turns out to be slow because iteration over cache involves a lot
> > of
> > > > random disk access for reading data pages referenced from leaf pages
> by
> > > > links.
> > > >
> > > > This is especially true when data is stored on disks with slow random
> > > > access, like SAS disks. In my case on modern SAS disks array reading
> > > speed
> > > > was like several MB/sec while sequential read speed in perf test was
> > > about
> > > > GB/sec.
> > > >
> > > > I was able to fix the issue by using ScanQuery with explicit
> partition
> > > set
> > > > and running simple warmup code before each partition scan.
> > > >
> > > > The code pins cold pages in memory in sequential order thus
> eliminating
> > > > random disk access. Speedup was like x100 magnitude.
> > > >
> > > > I suggest adding the improvement to the product's core  by always
> > > > sequentially preloading pages for all internal partition iterations
> > > (cache
> > > > iterators, scan queries, sql queries with scan plan) if partition is
> > cold
> > > > (low number of pinned pages).
> > > >
> > > > This also should speed up rebalancing from cold partitions.
> > > >
> > > > Ignite JIRA ticket [1]
> > > >
> > > > Thoughts ?
> > > >
> > > > [1] 

[GitHub] ignite pull request #4756: IGNITE-9487 Introduce IGNITE_REST_GETALL_AS_ARRAY...


2018-09-17

Thread
asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4756


---



[GitHub] ignite pull request #4771: Ignite 8454


2018-09-17

Thread
vveider
GitHub user vveider opened a pull request:

https://github.com/apache/ignite/pull/4771

Ignite 8454



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8454

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4771.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4771


commit 79312f9810ae17e8b6a0f095e7534086941dc0d5
Author: Ivanov Petr 
Date:   2018-05-14T08:05:54Z

IGNITE-8461 Scala-2.10 modules compilation failure under JDK 9

commit 9ce55087328377fd22ce6ccdc1cd2c6319f4aa0e
Author: Ivanov Petr 
Date:   2018-09-17T13:08:59Z

IGNITE-8454 Hadoop module compilation failure under JDK9




---



[GitHub] ignite pull request #4770: Ignite 7793


2018-09-17

Thread
Max-Pudov
GitHub user Max-Pudov opened a pull request:

https://github.com/apache/ignite/pull/4770

Ignite 7793



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7793

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4770.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4770


commit d769d939729e65fc3beb6a18f56d3f6d5291e51d
Author: Max-Pudov 
Date:   2018-09-17T12:58:07Z

IGNITE-7793 SQL does not work if value has sql field which name equals to 
affinity key name

commit 34c3b749b549a5eebd91ae5e92ad9fa33accaa39
Author: Max-Pudov 
Date:   2018-09-17T13:06:12Z

IGNITE-7793 SQL does not work if value has sql field which name equals to 
affinity key name




---



[GitHub] SomeFire opened a new pull request #11: IGNITE-9617 Different input possibilities for services page


2018-09-17

Thread
GitBox
SomeFire opened a new pull request #11: IGNITE-9617 Different input 
possibilities for services page
URL: https://github.com/apache/ignite-teamcity-bot/pull/11
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services



[jira] [Created] (IGNITE-9617) Different input possibilities for services page


2018-09-17

Thread
Ryabov Dmitrii (JIRA)
Ryabov Dmitrii created IGNITE-9617:
--

 Summary: Different input possibilities for services page
 Key: IGNITE-9617
 URL: https://issues.apache.org/jira/browse/IGNITE-9617
 Project: Ignite
  Issue Type: Sub-task
Reporter: Ryabov Dmitrii
Assignee: Ryabov Dmitrii


We should improve usability. Currently branch field requieres full TeamCity 
branch name like on index page. It should accept PR number too.

Same about JIRA ticket field. It should accept "IGNITE-XXX" and number only.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



[jira] [Created] (IGNITE-9616) SQL: Introduce H2 factory for holder of aggregate result


2018-09-17

Thread
Taras Ledkov (JIRA)
Taras Ledkov created IGNITE-9616:


 Summary: SQL: Introduce H2 factory for holder of aggregate result
 Key: IGNITE-9616
 URL: https://issues.apache.org/jira/browse/IGNITE-9616
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.6
Reporter: Taras Ledkov


H2 collects aggregate results at the simple HashMap (groups set and values set).
This causes an OOME error on large groups set and large group size with 
{{DISTINCT}}.

We have to introduce way to use our own implementation of the aggregate 
result's container.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



[GitHub] ignite pull request #4713: Ignite-9465: Node.js client: improve complex obje...


2018-09-17

Thread
asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4713


---



Re: loadcache very very slow


2018-09-17

Thread
Ilya Kasnacheev
Hello!

Please avoid writing questions about running Apache Ignite to developers
list. Developers list is suited for people who write Apache Ignite's code
or plan its features.

With regards to your project. I don't have MS SQL Server, and I especially
don't have your schema used to create tables.
I imagine that you might be missing index on key field in your table (the
one that Ignite will use according to your config) and thus it has to
resort to full scan while loading tables.

Regards,
-- 
Ilya Kasnacheev


пн, 17 сент. 2018 г. в 12:43, wt :

> hi Alexey
>
> When i debug it is like going down a maize. what i can see is that it
> enters
>
> class  = IgniteCacheProxyImpl
>
> void = loadCache
>which calls globalLoadCache
>which calls globalLoadCacheAsync
>
> then it goes all into all sorts of classes back and forth and fails because
> debugging breaks the connection timeout and the client is disconnected.
>
> here is the example project  Cluster-project(1).zip
> <
> http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/Cluster-project%281%29.zip>
>
>
> here is sample data data.csv
> 
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>



[GitHub] ignite pull request #4744: IGNITE-6454 reproducer fast finish


2018-09-17

Thread
dgarus
Github user dgarus closed the pull request at:

https://github.com/apache/ignite/pull/4744


---



Re: IgniteDataStreamer - into table and not key\value


2018-09-17

Thread
Ilya Kasnacheev


Do you have them package names in your config? ^^^ You have to remove those
when using simple name mapper.

If you're using those types from Java you will also need to turn on the
simple name mapper in Java config.

Regards
-- 
Ilya Kasnacheev


пн, 17 сент. 2018 г. в 15:07, wt :

> that is a typo, changing the name of the class to Indexcomposition doesn't
> make any difference. I also added the following to the config
>
>
>  
>  class="org.apache.ignite.configuration.BinaryConfiguration">
> 
>  class="org.apache.ignite.binary.BinaryBasicNameMapper">
> 
> 
> 
> 
> 
>
> the class names in .net are matching the java names in the config file
>
>
>  public class Indexcomposition
> {
>
> [QuerySqlField]
> public int vd { get; set; }
>
> [QuerySqlField]
> public double sharesinindex { get; set; }
>
> [QuerySqlField]
> public string name { get; set; }
>
> [QuerySqlField]
> public string isin { get; set; }
>
> [QuerySqlField]
> public string sedol { get; set; }
>
> [QuerySqlField]
> public string ric { get; set; }
>
> [QuerySqlField]
> public double close { get; set; }
>
> [QuerySqlField]
> public decimal rate { get; set; }
>
> }
>
>
>   
> .
>
> 
> 
>  value="java.lang.Integer"/>
>  value="java.lang.Double"/>
>  value="java.lang.String"/>
>  value="java.lang.String"/>
>  value="java.lang.String"/>
>  value="java.lang.String"/>
>  value="java.lang.Double"/>
>  value="java.math.BigDecimal"/>
> ...
> 
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>



Re: IgniteDataStreamer - into table and not key\value


2018-09-17

Thread
wt
that is a typo, changing the name of the class to Indexcomposition doesn't
make any difference. I also added the following to the config


 









the class names in .net are matching the java names in the config file


 public class Indexcomposition
{

[QuerySqlField]
public int vd { get; set; }

[QuerySqlField]
public double sharesinindex { get; set; }

[QuerySqlField]
public string name { get; set; }

[QuerySqlField]
public string isin { get; set; }

[QuerySqlField]
public string sedol { get; set; }

[QuerySqlField]
public string ric { get; set; }

[QuerySqlField]
public double close { get; set; }

[QuerySqlField]
public decimal rate { get; set; }

}


  
.











...





--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/



[GitHub] ignite pull request #4769: IGNITE-9501 Exchange latch optimizations


2018-09-17

Thread
Jokser
GitHub user Jokser opened a pull request:

https://github.com/apache/ignite/pull/4769

IGNITE-9501 Exchange latch optimizations



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9501

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4769.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4769


commit 416ea0ac7bf43fd6f0efa0c4f2c96245887ea50e
Author: Pavel Kovalenko 
Date:   2018-09-11T08:44:23Z

IGNITE-9501 WIP.

commit 980a006c451b20b574da458da4ff36a6bc303637
Author: Pavel Kovalenko 
Date:   2018-09-17T11:55:07Z

IGNITE-9501 Exclude joined nodes from exchange latch participants.




---



[jira] [Created] (IGNITE-9615) SQL: Client driver throws "Updates are not supported" exception


2018-09-17

Thread
Denis Mekhanikov (JIRA)
Denis Mekhanikov created IGNITE-9615:


 Summary: SQL: Client driver throws "Updates are not supported" 
exception
 Key: IGNITE-9615
 URL: https://issues.apache.org/jira/browse/IGNITE-9615
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.6
Reporter: Denis Mekhanikov
Assignee: Denis Mekhanikov


When calling certain methods on Connection object, retrieved from a client JDBC 
driver, {{SQLFeatureNotSupportedException}} with message "Updates are not 
supported" is thrown.

Affected methods:
 * {{JdbcConnection#setReadOnly(boolean)}}
 * {{JdbcConnection#prepareStatement(java.lang.String, int)}}
 * {{JdbcConnection#prepareStatement(java.lang.String, int[])}}
 * {{JdbcConnection#prepareStatement(java.lang.String, java.lang.String[])}}

{{SetReadOnly}} method shouldn't throw this exception, and exceptions thrown 
from {{prepareStatement}} methods should have a proper description.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



Re: IgniteDataStreamer - into table and not key\value


2018-09-17

Thread
Ilya Kasnacheev
Hello!

I don't see IndexHistory anywhere in your config file.

Most likely you need to make sure that binary object names for keys and
values match between your SQL, Java and .Net.
To do that, you would likely need to enable simple name binary mapping, and
make sure that all simple names are spelled identically.

Please see https://apacheignite-net.readme.io/docs/platform-interoperability

Regards,
-- 
Ilya Kasnacheev


пн, 17 сент. 2018 г. в 13:40, wt :

> hi Ilya
>
> Table initialised? it is a rmdbs integrated here is the config file
> Cluster-server.xml
> <
> http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/Cluster-server.xml>
>
>
> here is the class
>
> class IndexHistory
> {
>
> [QuerySqlField]
> public int VD { get; set; }
>
> [QuerySqlField]
> public double SHARESININDEX { get; set; }
>
> [QuerySqlField]
> public string NAME { get; set; }
>
> [QuerySqlField]
> public string ISIN { get; set; }
>
> [QuerySqlField]
> public string SEDOL { get; set; }
>
> [QuerySqlField]
> public string RIC { get; set; }
>
> [QuerySqlField]
> public double CLOSE { get; set; }
>
> [QuerySqlField]
> public decimal RATE { get; set; }
>
> }
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>



[jira] [Created] (IGNITE-9614) Java 9: Hadoop compilation error


2018-09-17

Thread
Peter Ivanov (JIRA)
Peter Ivanov created IGNITE-9614:


 Summary: Java 9: Hadoop compilation error
 Key: IGNITE-9614
 URL: https://issues.apache.org/jira/browse/IGNITE-9614
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.6
Reporter: Peter Ivanov
 Fix For: 2.7


When building Apache Ignite on TeamCity, the following error occurs
{code}
Could not resolve dependencies for project 
org.apache.ignite:ignite-hadoop:jar:2.7.0-SNAPSHOT: Could not find artifact 
jdk.tools:jdk.tools:jar:1.6 at specified path 
/usr/lib/jvm/java-9-oracle/../lib/tools.jar
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



Re: Using materialised views for queries with joins


2018-09-17

Thread
Sven Beauprez
Hi Dmitry,

Yes we can use those solutions in some cases, but not always. 

Replication is indeed the simplest, but sadly enough replicated caches are too 
much overhead. We have often a minimum of 12 nodes and all data must stay in 
sync 12x then. We do use it for small caches that don't need a lot of updates.

We use colocation all over the place. Colocation based on affinity keys is not 
possible though for distinct data sets with only some very specific 
relationships with _some other_ dataset, well known before hand.
 (eg. for example -not our exact use case which is more complex- items in a 
shopping basket with items from product inventory, both are in different caches 
managed on other nodes and it is not possible to denormalize such that the 
shopping basket knows the amount of availble items)


Regards,

Sven



 
SVEN BEAUPREZ 
 
L e a d   A r c h i t e c t

 

De Kleetlaan 5, B-1831 Diegem

www.theglue.com 

On 17/09/2018, 10:37, "Dmitriy Setrakyan"  wrote:

Hi Sven,

I will let others comment on the materialized view suggestion, but I have
another question.

*As we all know, joins are a nightmare in a distributed system*


Have you considered collocation or replication? If a table is replicated,
then it will be present on all the nodes and all joins will be fast. If two
partitioned tables are colocated based on some affinity key, then joins on
that affinity key will be fast as well.

Both, colocation and replication are supported by Ignite. Will any of these
approaches work for you?

D.

On Mon, Sep 17, 2018 at 11:04 AM Sven Beauprez 
wrote:

> All,
>
>
>
> We are in a situation where we have to query data over several caches. As
> we all know, joins are a nightmare in a distributed system and I know 
there
> are other means like denormalisation, but it is not sufficient anymore in
> some cases we have and we need the joins.
>
>
>
> We mainly work in an OLTP context, where queries are known in advance (ie
> dev time) and inpsired by following blog of several years ago, I was
> wondering if the concept of “materialized views” could make it into Apache
> Ignite.
>
> (
> 
https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
> )
>
>
>
> It would work as follows:
>
>- A query must register itself in Ignite at startup time (eg. via
>configuration) or during run time (eg. API call)
>- The registered query is parsed and a new “view” cache is created
>which will ‘cache’ the resultset of the query (could take a while, but
>intermediate status can be “warming up” and “hot” when ready)
>- All caches involved in the joins are now monitored for CUD
>operations and relevant data is stored in the new “view” cache so the 
view
>gets updated in real time
>- All operations must be ACID compliant
>- The view is queried via a very trivial select statement
>
>
>
> Would that be feasible as a new feature?
>
>
>
>
>
> Regards,
>
>
>
> Sven
>
>
>
>
>
>
>
> [image: cid:image001.png@01D3007B.4D007790]
>
>
>
> SVEN BEAUPREZ
>
>
>
> L e a d   A r c h i t e c t
>
>
>
> De Kleetlaan 5, B-1831 Diegem
>
> www.theglue.com
>





CRUD issues in Ignite Upsource


2018-09-17

Thread
Andrey Kuznetsov
Igniters,

I experience issues with Upsource [1]. Review creation/deletion or changes
of reviewer set in existing review lead to error message in a toast.
However, changes requested can be reflected some time later or after page
reload. Can anyone comment on this?

[1] https://reviews.ignite.apache.org/ignite

-- 
Best regards,
  Andrey Kuznetsov.



Re: Python thin client


2018-09-17

Thread
Igor Sapego
Dmitry,

Sorry, I was not clear enough. What I mean is that Ignite distributed
by both source and binary releases. Binary releases contain platforms
code, which is needed to write your own application in the language,
but does not contain developer stuff, such as tests, documentation
generating scripts, etc.

Of course, it is more common to use package managers when
developing your application, and of course, we are going to support
this approach. But still, we provide a way for a user to get any client
without any package manager.

I like the idea of supplying whl package in a binary release, though it
looks like it's going to take more effort to implement it. But I believe
except for the whl package, we will need to supply examples and
instructions, how user can run them.

What do you think?

Best Regards,
Igor


On Sat, Sep 15, 2018 at 7:03 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:

> Igor,
>
> I am in doubt here because I am not fully comprehend the meaning of
> "binary release". But if it is somehow related to the "distribution"
> thing, I would dare to suggest the following options:
>
> 1. Copy nothing. Just do
>
> ```
> $ python setup.py bdist_wheel
> $ twine upload dist/*
> ```
>
> during the build process (or separately) and let PyPI handle the
> distribution.
>
> This is the most natural and user-friendly way of distributing Python
> packages. End user might only do
>
> ```
> $ pip install pyignite
> ```
>
> as it is suggested by my readme file.
>
> 2. Supply the wheel package. It is the file 'pyignite-*.whl' from 'dist'
> directory, that should appear there as the "python setup.py bdist_wheel"
> command finishes. Actually, it can be combined with the first option.
>
> The wheel can then be installed by the end user:
>
> ```
> $ pip install pyignite-*.whl
> ```
>
> 3. Supply the following files:
>
> ignite/modules/platforms/python/pyignite/**
> ignite/modules/platforms/python/requirements/**
> ignite/modules/platforms/python/LICENSE
> ignite/modules/platforms/python/README.md
> ignite/modules/platforms/python/setup.py
>
> (** stands for "all the files and sub-folders recursively, starting from
> this folder".)
>
> It is not uncommon or wrong to distribute Python programs as text
> without prior packaging, but, in my personal experience, this is a way
> more suitable for one-time scripts or proprietary programs. For example,
> most of Python web apps relies on git for deployment, without any
> packaging subsystem.
>
> Here's a few words about wheels:
>
> https://wheel.readthedocs.io/
>
> Here's about uploading to PyPI with twine:
>
>
> https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives
>
> On 9/14/18 9:05 PM, Igor Sapego wrote:
> > Ok, good.
> >
> > Now, what is about installation? Which directories/files
> > need to be copied to ignite's binary release?
> >
> > Best Regards,
> > Igor
> >
>



Re: IgniteDataStreamer - into table and not key\value


2018-09-17

Thread
wt
hi Ilya

Table initialised? it is a rmdbs integrated here is the config file 
Cluster-server.xml

  

here is the class

class IndexHistory
{

[QuerySqlField]
public int VD { get; set; }

[QuerySqlField]
public double SHARESININDEX { get; set; }

[QuerySqlField]
public string NAME { get; set; }

[QuerySqlField]
public string ISIN { get; set; }

[QuerySqlField]
public string SEDOL { get; set; }

[QuerySqlField]
public string RIC { get; set; }

[QuerySqlField]
public double CLOSE { get; set; }

[QuerySqlField]
public decimal RATE { get; set; }

}



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/



Re: IgniteDataStreamer - into table and not key\value


2018-09-17

Thread
Ilya Kasnacheev
Hello!

Can you please show how your table is initialized? Also, can you show your
IndexHistory class?

Regards,
-- 
Ilya Kasnacheev


пн, 17 сент. 2018 г. в 13:33, wt :

> I am trying to load ignite using the data streamer into an ignite table.
> The
> streamer works but when querying the table it is empty. Upon further
> investigation it looks like the streamer is creating a key\value store in
> the same cache that the table resided. When querying the metadata it shows
> no changes to the cache. If i query the cache parsing in the surrogate key
> i
> injected into the code using the rest service i can see the records are
> there but in what looks like a key\value store. Can the IgniteDataStreamer
> load a table?
>
> here is the .net code
>
>try { connection.Open();
> command = new SqlCommand(sql, connection);
> dataReader = command.ExecuteReader();
> using (var ldr = ignite.GetDataStreamer IndexHistory>("IndexcompositionCache"))
> {
> ldr.PerNodeBufferSize = 1024;
> int i = 1;
> while (dataReader.Read())
> {
> var o = new IndexHistory()
> {
> VD = (int)dataReader.GetValue(0),
> SHARESININDEX = (double)dataReader.GetValue(1),
> NAME = (string)dataReader.GetValue(2),
> ISIN = (string)dataReader.GetValue(3),
> SEDOL = (string)dataReader.GetValue(4),
> RIC = (string)dataReader.GetValue(5),
> CLOSE = (double)dataReader.GetValue(6),
> RATE = (decimal)dataReader.GetValue(7)
> };
> ldr.AddData(i, o );
>
> i = i + 1;
> }
>
> }
> dataReader.Close();
> command.Dispose();
> connection.Close(); }
> catch (Exception ex)
> {
> Console.WriteLine(ex.Message.ToString());
> }
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>



IgniteDataStreamer - into table and not key\value


2018-09-17

Thread
wt
I am trying to load ignite using the data streamer into an ignite table. The
streamer works but when querying the table it is empty. Upon further
investigation it looks like the streamer is creating a key\value store in
the same cache that the table resided. When querying the metadata it shows
no changes to the cache. If i query the cache parsing in the surrogate key i
injected into the code using the rest service i can see the records are
there but in what looks like a key\value store. Can the IgniteDataStreamer 
load a table?

here is the .net code

   try { connection.Open();
command = new SqlCommand(sql, connection);
dataReader = command.ExecuteReader();
using (var ldr = ignite.GetDataStreamer("IndexcompositionCache"))
{
ldr.PerNodeBufferSize = 1024;
int i = 1;
while (dataReader.Read())
{
var o = new IndexHistory()
{
VD = (int)dataReader.GetValue(0),
SHARESININDEX = (double)dataReader.GetValue(1),
NAME = (string)dataReader.GetValue(2),
ISIN = (string)dataReader.GetValue(3),
SEDOL = (string)dataReader.GetValue(4),
RIC = (string)dataReader.GetValue(5),
CLOSE = (double)dataReader.GetValue(6),
RATE = (decimal)dataReader.GetValue(7)
};
ldr.AddData(i, o );

i = i + 1;
}

}
dataReader.Close();
command.Dispose();
connection.Close(); }
catch (Exception ex)
{ 
Console.WriteLine(ex.Message.ToString());
}



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/



[GitHub] ignite pull request #4660: IGNITE-9320: Mvcc cache configuration and cluster...


2018-09-17

Thread
asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4660


---



[GitHub] ignite pull request #4759: IGNITE-9599 Add ability to manage compression lev...


2018-09-17

Thread
a-polyakov
Github user a-polyakov closed the pull request at:

https://github.com/apache/ignite/pull/4759


---



Re: Cache scan efficiency


2018-09-17

Thread
Maxim Muzafarov
Folks,

Such warming up can be an effective technique for performing calculations
which required large cache
data reads, but I think it's the single narrow use case of all over Ignite
store usages. Like all other
powerfull techniques, we should use it wisely. In the general case, I think
we should consider other
techniques mentioned by Vladimir and may create something like `global
statistics of cache data usage`
to choose the best technique in each case.

For instance, it's not obvious what would take longer: multi-block reads or
50 single-block reads issues
sequentially. It strongly depends on used hardware under the hood and might
depend on workload system
resources (CPU-intensive calculations and I\O access) as well. But
`statistics` will help us to choose
the right way.


On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov  wrote:

> Hi Alexei,
>
> I did not find any PRs associated with the ticket for check code changes
> behind this idea. Are there any PRs?
>
> If we create some forwards scan of pages, it should be a very intellectual
> algorithm including a lot of parameters (how much RAM is free, how probably
> we will need next page, etc). We had the private talk about such idea some
> time ago.
>
> By my experience, Linux systems already do such forward reading of file
> data (for corresponding sequential flagged file descriptors), but some
> prefetching of data at the level of application may be useful for O_DIRECT
> file descriptors.
>
> And one more concern from me is about selecting a right place in the system
> to do such prefetch.
>
> Sincerely,
> Dmitriy Pavlov
>
> вс, 16 сент. 2018 г. в 19:54, Vladimir Ozerov :
>
> > HI Alex,
> >
> > This is good that you observed speedup. But I do not think this solution
> > works for the product in general case. Amount of RAM is limited, and
> even a
> > single partition may need more space than RAM available. Moving a lot of
> > pages to page memory for scan means that you evict a lot of other pages,
> > what will ultimately lead to bad performance of subsequent queries and
> > defeat LRU algorithms, which are of great improtance for good database
> > performance.
> >
> > Database vendors choose another approach - skip BTrees, iterate direclty
> > over data pages, read them in multi-block fashion, use separate scan
> buffer
> > to avoid excessive evictions of other hot pages. Corresponding ticket for
> > SQL exists [1], but idea is common for all parts of the system, requiring
> > scans.
> >
> > As far as proposed solution, it might be good idea to add special API to
> > "warmup" partition with clear explanation of pros (fast scan after
> warmup)
> > and cons (slowdown of any other operations). But I think we should not
> make
> > this approach part of normal scans.
> >
> > Vladimir.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-6057
> >
> >
> > On Sun, Sep 16, 2018 at 6:44 PM Alexei Scherbakov <
> > alexey.scherbak...@gmail.com> wrote:
> >
> > > Igniters,
> > >
> > > My use case involves scenario where it's necessary to iterate over
> > > large(many TBs) persistent cache doing some calculation on read data.
> > >
> > > The basic solution is to iterate cache using ScanQuery.
> > >
> > > This turns out to be slow because iteration over cache involves a lot
> of
> > > random disk access for reading data pages referenced from leaf pages by
> > > links.
> > >
> > > This is especially true when data is stored on disks with slow random
> > > access, like SAS disks. In my case on modern SAS disks array reading
> > speed
> > > was like several MB/sec while sequential read speed in perf test was
> > about
> > > GB/sec.
> > >
> > > I was able to fix the issue by using ScanQuery with explicit partition
> > set
> > > and running simple warmup code before each partition scan.
> > >
> > > The code pins cold pages in memory in sequential order thus eliminating
> > > random disk access. Speedup was like x100 magnitude.
> > >
> > > I suggest adding the improvement to the product's core  by always
> > > sequentially preloading pages for all internal partition iterations
> > (cache
> > > iterators, scan queries, sql queries with scan plan) if partition is
> cold
> > > (low number of pinned pages).
> > >
> > > This also should speed up rebalancing from cold partitions.
> > >
> > > Ignite JIRA ticket [1]
> > >
> > > Thoughts ?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-8873
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
-- 
--
Maxim Muzafarov



[jira] [Created] (IGNITE-9613) MVCC: Clean up code related to resolved tickets.


2018-09-17

Thread
Andrew Mashenkov (JIRA)
Andrew Mashenkov created IGNITE-9613:


 Summary: MVCC: Clean up code related to resolved tickets.
 Key: IGNITE-9613
 URL: https://issues.apache.org/jira/browse/IGNITE-9613
 Project: Ignite
  Issue Type: Improvement
Reporter: Andrew Mashenkov


For now we have a number of TODOs that should be cleaned after IGNITE-6938 and 
IGNITE-6739 complete.

We have to review these TODOs and either remove or create a tickets for them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



[jira] [Created] (IGNITE-9612) Improve checkpoint mark phase speed.


2018-09-17

Thread
Alexei Scherbakov (JIRA)
Alexei Scherbakov created IGNITE-9612:
-

 Summary: Improve checkpoint mark phase speed.
 Key: IGNITE-9612
 URL: https://issues.apache.org/jira/browse/IGNITE-9612
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexei Scherbakov
Assignee: Alexei Scherbakov
 Fix For: 2.7


I'm observing regular slow checkpoints due to long mark duration, which is not 
related to dirty pages number:

{noformat}
2018-09-01 14:55:20.408 [INFO 
][db-checkpoint-thread-#241%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.p.GridCacheDatabaseSharedManager]
 Checkpoint started [checkpointId=01e0c7bf-842f-4ed6-8589-b4904063434f, 
startPtr=FileWALPointer [idx=19814, fileOff=948996096, len=5233457],
checkpointLockWait=0ms, checkpointLockHoldTime=951ms, 
walCpRecordFsyncDuration=39ms, pages=78477, reason='timeout']
2018-09-01 14:55:21.307 [INFO 
][db-checkpoint-thread-#241%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.p.GridCacheDatabaseSharedManager]
 Checkpoint finished [cpId=01e0c7bf-842f-4ed6-8589-b4904063434f, pages=78477, 
markPos=FileWALPointer [idx=19814, fileOff=948996096, len=5233457], 
walSegmentsCleared=0, walSegmentsCovered=[], *markDuration=1002m*s, 
pagesWrite=478ms, fsync=421ms, total=1901ms] 
{noformat}

{noformat}
2018-09-01 14:58:20.355 [INFO 
][db-checkpoint-thread-#241%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.p.GridCacheDatabaseSharedManager]
 Checkpoint started [checkpointId=09d1f4bc-d3f3-4a16-b291-89d7fa745ea5, 
startPtr=FileWALPointer [idx=19814, fileOff=124208, len=5233457], 
checkpointLockWait=0ms, checkpointLockHoldTime=926ms, 
walCpRecordFsyncDuration=14ms, pages=10837, reason='timeout']
2018-09-01 14:58:20.480 [INFO 
][db-checkpoint-thread-#241%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.p.GridCacheDatabaseSharedManager]
 Checkpoint finished [cpId=09d1f4bc-d3f3-4a16-b291-89d7fa745ea5, pages=10837, 
markPos=FileWALPointer [idx=19814, fileOff=124208, len=5233457], 
walSegmentsCleared=0, walSegmentsCovered=[], *markDuration=943ms*, 
pagesWrite=64ms, fsync=61ms, total=1068ms]
{noformat}

Debugging has revealed what this is due to large amount of work required to 
save metadata for metapages and free/reuse lists. Because this is done under 
checkpoint write lock, all other activities are blocked, resulting in increased 
tx and atomic ops latency.

Simple solution: parallelize metadata processing during mark phase.

Best way to solve the problem is described in IGNITE-9520.






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



[jira] [Created] (IGNITE-9611) OutOfMemory error in Cache 2 test sute


2018-09-17

Thread
Alexey Stelmak (JIRA)
Alexey Stelmak created IGNITE-9611:
--

 Summary: OutOfMemory error in Cache 2 test sute
 Key: IGNITE-9611
 URL: https://issues.apache.org/jira/browse/IGNITE-9611
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Stelmak
Assignee: Alexey Stelmak






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



Re: loadcache very very slow


2018-09-17

Thread
wt
hi Alexey

When i debug it is like going down a maize. what i can see is that it enters 

class  = IgniteCacheProxyImpl

void = loadCache 
   which calls globalLoadCache
   which calls globalLoadCacheAsync

then it goes all into all sorts of classes back and forth and fails because
debugging breaks the connection timeout and the client is disconnected.

here is the example project  Cluster-project(1).zip

  

here is sample data data.csv
  





--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/



Re: how to use kafka 2.0 in ignite-kafka


2018-09-17

Thread
kcheng.mvp
Thank you very much!
 I just wondering is there any way to decouple with the kafka version. As
right now it was bound tightly api call directly.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/



Re: loadcache very very slow


2018-09-17

Thread
Alexey Kuznetsov
Hi,

As I see from code of CacheAbstractJdbcStore
it will try to use range query if possible.

see: CacheAbstractJdbcStore#loadCacheRange

As you are using SQL Server make sure that SQLServerDialect was set to
your  store, please provide config generated by Web Console.

Or just set a break point and trace down to
CacheAbstractJdbcStore#loadCacheRange and see what it is doing actually.

Or attach a reproducer here (generate project + sample SQL dump of your
data, do not forget to replace sensitive data by random stuff).

On Mon, Sep 17, 2018 at 3:56 PM wt  wrote:

> Hi
>
> I have a project that is generated by the web console (2.5) and it is
> loading an integrated db (SQL Server) table that has 4.5 million rows. When
> running the load cache it is loading 1-3 records per second which is
> unacceptable. Looking at the documentation i see examples of the overridden
> load cache mechanism which makes me think that the default loading
> mechanism
> in this project is not optimal for bulk loads. If i run a query on the jdbc
> db driver it is returning 100k of rows per second.
>
> Looking for advice here as i want to maintain the strategy of not needing
> to
> write load code but obviously want something that wont take forever to
> load.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


-- 
Alexey Kuznetsov



[GitHub] ignite pull request #3876: IGNITE-8092 Put hang on destroy.


2018-09-17

Thread
xtern
Github user xtern closed the pull request at:

https://github.com/apache/ignite/pull/3876


---



[GitHub] ignite pull request #4095: IGNITE-7618 validateCache synchronously waits for...


2018-09-17

Thread
SomeFire
Github user SomeFire closed the pull request at:

https://github.com/apache/ignite/pull/4095


---



loadcache very very slow


2018-09-17

Thread
wt
Hi

I have a project that is generated by the web console (2.5) and it is
loading an integrated db (SQL Server) table that has 4.5 million rows. When
running the load cache it is loading 1-3 records per second which is
unacceptable. Looking at the documentation i see examples of the overridden
load cache mechanism which makes me think that the default loading mechanism
in this project is not optimal for bulk loads. If i run a query on the jdbc
db driver it is returning 100k of rows per second. 

Looking for advice here as i want to maintain the strategy of not needing to
write load code but obviously want something that wont take forever to load. 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/



[GitHub] ignite pull request #4768: IGNITE-9610 IgniteStandByClusterTest.testSimple i...


2018-09-17

Thread
ibessonov
GitHub user ibessonov opened a pull request:

https://github.com/apache/ignite/pull/4768

IGNITE-9610 IgniteStandByClusterTest.testSimple is failed in master



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9610

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4768.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4768


commit ede5fe42934f0e9f59d0dfa454b1d5d937d2bdc0
Author: ibessonov 
Date:   2018-09-17T08:42:32Z

IGNITE-9610 Supposed fix of data race that caused 
IgniteStandByClusterTest#testSimple to fail




---



Re: Using materialised views for queries with joins


2018-09-17

Thread
Dmitriy Setrakyan
Hi Sven,

I will let others comment on the materialized view suggestion, but I have
another question.

*As we all know, joins are a nightmare in a distributed system*


Have you considered collocation or replication? If a table is replicated,
then it will be present on all the nodes and all joins will be fast. If two
partitioned tables are colocated based on some affinity key, then joins on
that affinity key will be fast as well.

Both, colocation and replication are supported by Ignite. Will any of these
approaches work for you?

D.

On Mon, Sep 17, 2018 at 11:04 AM Sven Beauprez 
wrote:

> All,
>
>
>
> We are in a situation where we have to query data over several caches. As
> we all know, joins are a nightmare in a distributed system and I know there
> are other means like denormalisation, but it is not sufficient anymore in
> some cases we have and we need the joins.
>
>
>
> We mainly work in an OLTP context, where queries are known in advance (ie
> dev time) and inpsired by following blog of several years ago, I was
> wondering if the concept of “materialized views” could make it into Apache
> Ignite.
>
> (
> https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
> )
>
>
>
> It would work as follows:
>
>- A query must register itself in Ignite at startup time (eg. via
>configuration) or during run time (eg. API call)
>- The registered query is parsed and a new “view” cache is created
>which will ‘cache’ the resultset of the query (could take a while, but
>intermediate status can be “warming up” and “hot” when ready)
>- All caches involved in the joins are now monitored for CUD
>operations and relevant data is stored in the new “view” cache so the view
>gets updated in real time
>- All operations must be ACID compliant
>- The view is queried via a very trivial select statement
>
>
>
> Would that be feasible as a new feature?
>
>
>
>
>
> Regards,
>
>
>
> Sven
>
>
>
>
>
>
>
> [image: cid:image001.png@01D3007B.4D007790]
>
>
>
> SVEN BEAUPREZ
>
>
>
> L e a d   A r c h i t e c t
>
>
>
> De Kleetlaan 5, B-1831 Diegem
>
> www.theglue.com
>



[jira] [Created] (IGNITE-9610) IgniteStandByClusterTest.testSimple is broken in master


2018-09-17

Thread
Ivan Bessonov (JIRA)
Ivan Bessonov created IGNITE-9610:
-

 Summary: IgniteStandByClusterTest.testSimple is broken in master
 Key: IGNITE-9610
 URL: https://issues.apache.org/jira/browse/IGNITE-9610
 Project: Ignite
  Issue Type: Test
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


Test was broken by the IGNITE-7618 fix



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)



Using materialised views for queries with joins


2018-09-17

Thread
Sven Beauprez
All,

We are in a situation where we have to query data over several caches. As we 
all know, joins are a nightmare in a distributed system and I know there are 
other means like denormalisation, but it is not sufficient anymore in some 
cases we have and we need the joins.

We mainly work in an OLTP context, where queries are known in advance (ie dev 
time) and inpsired by following blog of several years ago, I was wondering if 
the concept of “materialized views” could make it into Apache Ignite.
(https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
 )

It would work as follows:

  *   A query must register itself in Ignite at startup time (eg. via 
configuration) or during run time (eg. API call)
  *   The registered query is parsed and a new “view” cache is created which 
will ‘cache’ the resultset of the query (could take a while, but intermediate 
status can be “warming up” and “hot” when ready)
  *   All caches involved in the joins are now monitored for CUD operations and 
relevant data is stored in the new “view” cache so the view gets updated in 
real time
  *   All operations must be ACID compliant
  *   The view is queried via a very trivial select statement

Would that be feasible as a new feature?


Regards,

Sven



[cid:image001.png@01D3007B.4D007790]

SVEN BEAUPREZ

L e a d   A r c h i t e c t

De Kleetlaan 5, B-1831 Diegem
www.theglue.com



Re: how to use kafka 2.0 in ignite-kafka


2018-09-17

Thread
Dmitriy Pavlov
Hi, Actually there is no such plan for 2.7. In 2.7 we will probably switch
to 1.1.1.

Sincerely,
Dmitriy Pavlov

пн, 17 сент. 2018 г. в 5:22, kcheng.mvp :

> Is there any plan to migrate kafka 2.0 in ignite-kafka?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>




    

      
    
    
96 matches
    

    
Advanced search


Search the list




    
    

      

                               Site Navigation
      
      

        
The Mail Archive home
        

          dev - all messages
        

          dev  - about the list
        
Expand
      
    

    

      

  				Mail list logo
      
      
    
  

    

	        	      Footer information
    
    

      
The Mail Archive home
      
Add your mailing list
      
FAQ
      
Support
      
Privacy
    
  
#
Id
Total 
tests
Failed tests
Ignored tests
Muted tests
Total issues
Total run 
time