[jira] [Created] (IGNITE-10665) CassandraSessionImpl creates IgniteException even if it is not needed

[Dmitry Konstantinov (JIRA)]

Dmitry Konstantinov created IGNITE-10665:


 Summary: CassandraSessionImpl creates IgniteException even if it 
is not needed
 Key: IGNITE-10665
 URL: https://issues.apache.org/jira/browse/IGNITE-10665
 Project: Ignite
  Issue Type: Bug
  Components: cassandra
Affects Versions: 2.7, 2.6, 2.5, 2.4
Reporter: Dmitry Konstantinov


* 
[https://github.com/apache/ignite/blob/master/modules/cassandra/store/src/main/java/org/apache/ignite/cache/store/cassandra/session/CassandraSessionImpl.java#L208]
 * 
[https://git.netcracker.com/thirdparty.modified/ignite/blob/2.4.0_nc_modified/modules/cassandra/store/src/main/java/org/apache/ignite/cache/store/cassandra/session/CassandraSessionImpl.java#L399]
 

Example from a thread dump:
{code:java}
"flusher-5-#97" #126 prio=5 os_prio=0 tid=0x7fc070228000 nid=0x2a68 
runnable [0x7fbff02a3000]
   java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
- locked <0x00076b2201d0> (a org.apache.ignite.IgniteException)
at java.lang.Throwable.(Throwable.java:265)
at java.lang.Exception.(Exception.java:66)
at java.lang.RuntimeException.(RuntimeException.java:62)
at org.apache.ignite.IgniteException.(IgniteException.java:44)
at 
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:209)
at 
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.writeAll(CassandraCacheStore.java:354)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.updateStore(GridCacheWriteBehindStore.java:814)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.applyBatch(GridCacheWriteBehindStore.java:726)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.access$2400(GridCacheWriteBehindStore.java:76)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.flushCacheCoalescing(GridCacheWriteBehindStore.java:1143)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.body(GridCacheWriteBehindStore.java:1016)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)

{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

CONTRIBUTING.md first version

[Dmitriy Pavlov]

Hi Igniters,

Just to inform, the very first version of the Contribution step by step
guide was added to GitHub. It is one more entry point for newcomers to find
out what and how should be done to contribute:
https://github.com/apache/ignite/blob/master/CONTRIBUTING.md#contributing-to-apache-ignite


Feel free to share with newcomers and contribute.

Sincerely,
Dmitriy Pavlov

[jira] [Created] (IGNITE-10662) IgnitePdsDataRegionMetricsTest.testMemoryUsageMultipleNodes fails in TX mode.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-10662:
-

 Summary: 
IgnitePdsDataRegionMetricsTest.testMemoryUsageMultipleNodes fails in TX mode.
 Key: IGNITE-10662
 URL: https://issues.apache.org/jira/browse/IGNITE-10662
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Reporter: Andrew Mashenkov


IgnitePdsDataRegionMetricsTest.testMemoryUsageMultipleNodes fails with 
TRANSACTIONAL cache.

We should have this test for both Atomic and Tx mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ML][DISCUSSION] ML Package reorganization

[Alexey Zinoviev]

Support idea with postpone!

ср, 12 дек. 2018 г. в 15:26, Yuriy Babak :

> Dmitry,
>
> Yes, those changes will affect users. But we could postpone this change to
> the next major release(3.0) and we could implement a simple migration tool
> for the ML part.
>
> Regards,
> Yuriy
>
> ср, 12 дек. 2018 г. в 15:10, Dmitriy Pavlov :
>
> > Hi Yuri,
> >
> > Would it affect users?
> >
> > Ignite-code has separation of API classes (not internal) and internal
> > stuff. Internal classes may be changed from version to version, but API
> > can't be renamed/moved/and so on. What about ML?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 11 дек. 2018 г. в 16:56, Yuriy Babak :
> >
> > > Igniters,
> > >
> > > I want to discuss package structure for ML module. Actually I don't
> like
> > > our current package organization. The main problem is that we have lots
> > of
> > > different packages under root package.
> > >
> > > For examples algorithm related packages on same level like some utility
> > > infrastructure packages such as environment(package with some classes
> > > related with training environment).
> > >
> > > So I created branch [1] with example of new package structure. Please
> > share
> > > any feedback about this package structure and about this idea itself.
> > >
> > > Regards,
> > > Yuriy.
> > >
> > > [1] -
> > >
> > >
> >
> https://github.com/YuriBabak/ignite/tree/ignite-10641/modules/ml/src/main/java/org/apache/ignite/ml
> > >
> >
>

[jira] [Created] (IGNITE-10661) Web Console: E2E tests failed because of problems with Docker file

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-10661:
-

 Summary: Web Console: E2E tests failed because of problems with 
Docker file
 Key: IGNITE-10661
 URL: https://issues.apache.org/jira/browse/IGNITE-10661
 Project: Ignite
  Issue Type: Improvement
  Components: wizards
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov
 Fix For: 2.8


Build failed with
{code}
[20:47:59][Step 3/4] npm ERR! code EADDRNOTAVAIL
[20:47:59][Step 3/4] npm ERR! errno EADDRNOTAVAIL
[20:47:59][Step 3/4] npm ERR! request to 
https://registry.npmjs.org/@babel%2fplugin-transform-parameters failed, reason: 
getaddrinfo EADDRNOTAVAIL registry.npmjs.org registry.npmjs.org:443
[20:47:59][Step 3/4] 
[20:47:59][Step 3/4] npm ERR! A complete log of this run can be found in:
[20:47:59][Step 3/4] npm ERR! 
/root/.npm/_logs/2018-12-12T13_47_59_616Z-debug.log
[20:48:00][Step 3/4] Service 'unit_tests' failed to build: The command '/bin/sh 
-c npm install --no-optional' returned a non-zero code: 1
{code}

It is known issue with alpine:edge Docker image



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10664) need to make identical the behavior of control.sh and control.bat

[ARomantsov (JIRA)]

ARomantsov created IGNITE-10664:
---

 Summary:  need to make identical the behavior of control.sh and 
control.bat
 Key: IGNITE-10664
 URL: https://issues.apache.org/jira/browse/IGNITE-10664
 Project: Ignite
  Issue Type: Bug
  Components: clients
Affects Versions: 2.7
 Environment: windows, nix
Reporter: ARomantsov
 Fix For: 2.8


Now control.bat contain next line: 
if not "%NO_PAUSE%" == "1" pause 
and after execution it ready to press button.

on the other hand - control.sh not contain any pause



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10663) Consistency check

[Anton Vinogradov (JIRA)]

Anton Vinogradov created IGNITE-10663:
-

 Summary: Consistency check
 Key: IGNITE-10663
 URL: https://issues.apache.org/jira/browse/IGNITE-10663
 Project: Ignite
  Issue Type: Task
Reporter: Anton Vinogradov
Assignee: Anton Vinogradov


Need to provide API to make sure that all data nodes contain the same value per 
key.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ML][DISCUSSION] ML Package reorganization

[aplatonov]

Yes, I totally agree with you. But I think we should move "composition"
package to "algorithm"  because of classes of "composition" are
meta-algothms. Class ModelComposition with "aggregator"-package can be moved
to "model". Also, we should refactor Dataset-classes. There are two
different Dataset.java files at least.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

newbie to Apache Ignite.

[Lawrence.Eng]

Hi,
   I just started looking to adapt Apache Ignite into our project as an 
in-memory distributed cache and potentially as distributed computing grid.
   An Ignite client application, ClientPutGetExample, encountered "Ignite 
cluster is unavailable" error intermittently.Ignite 2.7.0 was being
   used in the dev environment.There was  a cluster of 2 Ignite servers 
which was running with the client application in the same host.
   I noticed that the error was encountered after one of the server was stopped.

   How would I resolve the issue?
   Thank you for your help in advance.

Regards,
Lawrence





_

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

__
If you are incorporated or operating in Australia, please see 
https://www.home.barclays/disclosures/importantapacdisclosures.html for 
important disclosure.
__
__
How we use personal information  see our privacy notice 
https://www.investmentbank.barclays.com/disclosures/personalinformationuse.html
_




http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="
   http://www.springframework.org/schema/beans
   http://www.springframework.org/schema/beans/spring-beans.xsd;>



   

   
   
	   
	   
	   
	   

   

   
  
   
	  
   

Re: CONTRIBUTING.md first version

[Dmitriy Pavlov]

There is nothing wrong with wiki (excepting it is sometimes way too slow
and it may be annoying ).

Github markdown is a way to
1. Keep documenting under version control system
2. Github recommends to provide this doc in the standard place.

Of course, we can move this new doc content to wiki and just provide link
from contributing.md to wiki. Should we?

I can also add a copy to a wiki. Denis, folks,  please share your vision.

чт, 13 дек. 2018 г., 0:32 Denis Magda :

> What is wrong with Ignite wiki? It seems like the right place for the doc
> like this.
>
> --
> Denis
>
> On Wed, Dec 12, 2018 at 9:21 AM Dmitriy Pavlov  wrote:
>
> > Hi Igniters,
> >
> > Just to inform, the very first version of the Contribution step by step
> > guide was added to GitHub. It is one more entry point for newcomers to
> find
> > out what and how should be done to contribute:
> >
> >
> https://github.com/apache/ignite/blob/master/CONTRIBUTING.md#contributing-to-apache-ignite
> >
> >
> > Feel free to share with newcomers and contribute.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
>

Re: CONTRIBUTING.md first version

[Denis Magda]

What is wrong with Ignite wiki? It seems like the right place for the doc
like this.

--
Denis

On Wed, Dec 12, 2018 at 9:21 AM Dmitriy Pavlov  wrote:

> Hi Igniters,
>
> Just to inform, the very first version of the Contribution step by step
> guide was added to GitHub. It is one more entry point for newcomers to find
> out what and how should be done to contribute:
>
> https://github.com/apache/ignite/blob/master/CONTRIBUTING.md#contributing-to-apache-ignite
>
>
> Feel free to share with newcomers and contribute.
>
> Sincerely,
> Dmitriy Pavlov
>

[jira] [Created] (IGNITE-10666) Incorrect value in near cache after concurrent client get and update from server

[Semen Boikov (JIRA)]

Semen Boikov created IGNITE-10666:
-

 Summary: Incorrect value in near cache after concurrent client get 
and update from server
 Key: IGNITE-10666
 URL: https://issues.apache.org/jira/browse/IGNITE-10666
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.7
Reporter: Semen Boikov
Assignee: Semen Boikov
 Fix For: 2.8


Tests GridNearCacheStoreUpdateTest  
testTransactionAtomicUpdateNear/testAtomicUpdateNear  sometimes fail: after 
concurrent get from client with near cache and put from server, sometimes near 
cache contains outdated value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Pre-touch for Ignite off-heap memory

[Павлухин Иван]

Hi Stan,

As I participated in discussion in current thread I would like to
leave a comment.

Your concerns looks clear for me and if you believe that pre-touch
will help product users then I have nothing against it.
ср, 12 дек. 2018 г. в 11:09, Nikolay Izhikov :
>
> Hello, Stanislav.
>
> As far as I can see, we have controversal ticket based on previous dev-list 
> discussion:
>
> IGNITE-9113 - Allocate memory for a data region when first cache assigned to 
> this region is created [1]
>
> I planned to implement it soon.
> Looks like we should have several options of memory(data regions) allocation:
>
> - allocate all on startup (AFAIK this is how current implementation 
> behave)
> - allocate all on startup AND pre touch.
> - allocate specific data region for first assignment.
> - allocate specific data region for first assignment AND pre touch.
>
> What do you think?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9113
>
> В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> > Igniters,
> >
> > What is being suggested here is an Ignite off-heap’s version of Java’s 
> > -XX:+AlwaysPreTouch.
> > The latter is known to be used to guarantee that the committed memory is 
> > backed by physical RAM.
> > This ensures that
> > a) JVM doesn’t have to do it during the actual work (avoiding overhead for 
> > physical page allocation, possible contention with page cache, etc)
> > b) JVM fails fast if the Xmx is greater than available RAM
> > c) New processes will not be able to claim the memory JVM took for itself
> >
> > Currently one can’t get the same benefits for Ignite because we use 
> > off-heap as well as heap.
> > So, we can implement a similar feature for Ignite – and make sure the users 
> > can get all the memory pre-touch benefits if they want.
> > Of course, it impacts startup time so we should enable it by a 
> > configuration property (I’d suggest a system property for now).
> >
> > Are there any objections to implementing this?
> >
> > Thanks,
> > Stan
> >
> > From: Павлухин Иван
> > Sent: 31 октября 2018 г. 12:50
> > To: dev@ignite.apache.org
> > Subject: Re: Pre-touch for Ignite off-heap memory
> >
> > Hi,
> >
> > I did an experiment described above. I created a patch with pre-touch [1]
> > and started a JVM with an option -XX:+AlwaysPreTouch and configured
> > Ignite with equal values for initial and max sizes for each data region.
> > I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
> > to observe JVM OOM-killed.
> >
> > [1] https://github.com/apache/ignite/pull/5220
> > [2]
> > https://gist.github.com/pavlukhin/e5e6605e9b4366627ba8d1aab42f#file-hs_err_pid5763-log
> > [3]
> > https://gist.github.com/pavlukhin/e5e6605e9b4366627ba8d1aab42f#file-hs_err_pid6411-log
> >
> > вт, 30 окт. 2018 г. в 9:19, Павлухин Иван :
> >
> > > Hi guys,
> > >
> > > I am not aware that it is possible to run JVM in "allocation-free" 
> > > fashion.
> > > If you know that it is possible please share it. As I know JVM allocates
> > > memory out of garbage collectable area for internal purposes like JIT,
> > > GC itself. Also native built-it code can request memory allocation from 
> > > OS,
> > > e.g. zip facilities. If we imagine that system is running under memory
> > > allocation
> > > which is close to a limit then even small allocation request can fail and
> > > lead
> > > to OOM killing.
> > >
> > > But I think that a simple and useful thing that could be done first is
> > > making
> > > an experiment. As Andrey mentioned
> > > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > >
> > > region
> > > > MAX size equal to MIN and get region allocated on node start.
> > >
> > > If it is so then it should not be hard to launch Ignite and observe it
> > > running
> > > very close to OS memory limit with Java heap and Ignite off-heap
> > > pre-touched.
> > > After that one could check whether it is possible to observe Ignite OOM
> > > killed.
> > > Let's say my bet is that it is relatively easy to catch OOM here.
> > >
> > > What do you think?
> > >
> > > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov :
> > >
> > > > Andrey,
> > > >
> > > > Probability of a OOM kill will be much lower if offheap is pretouched.
> > > > What
> > > > do you mean by JVM internal needs? In my understanding if user enables
> > > > option to pretouch heap and fixes the heap to prevent jvm releasing 
> > > > memory
> > > > back to OS, then OOM killing is very unlikely.
> > > >
> > > > I would agree that pretouch for offheap may be helpful in many cases.
> > > >
> > > > --Yakov
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
> >



-- 
Best regards,
Ivan Pavlukhin

Re: [DISCUSSION] Relocation of Apache git repositories from git-wip-us.apache.org to GitBox

[Dmitriy Pavlov]

Folks, Incubator finished its migration to GitBox -
https://gitbox.apache.org/repos/asf?p=incubator.git

Because it seems majority don't care too much about migration I'm going to
consider this decision approved by silence and could be made by lazy
consensus in 72 hours.


вт, 11 дек. 2018 г. в 19:25, Alexey Goncharuk :

> Given that there is no option to stay on the old repository and
> mass-migration is scheduled for Feb, 7th, I think it is better to prepare
> and move the repository before the date.
>
> As far as I understand, from developers standpoint we only need to change
> the git remote entry. Petr, Sergey, can you asses what changes are need to
> be done in order to support this migration for TC and release procedure?
>
> вт, 11 дек. 2018 г. в 19:20, Dmitriy Pavlov :
>
> > Hi All,
> >
> > The Apache Ignite still has a repository on git-wip-us for its code, as
> > well, for release.
> >
> > Does anyone have a problem with moving over the Ignite git-wip-us
> > repository to gitbox voluntarily? This means integrated access and easy
> PRs
> > (write access to the GitHub repo).
> >
> > We need to document support for the decision from a mailing list post, so
> > here it is.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 8 дек. 2018 г. в 00:16, Dmitriy Pavlov :
> >
> > > Hi Igniters,
> > >
> > > What do you think about the voluntary migration of project repositories
> > > from git-wip to GitBox? (See details in the forwarded email.)
> > >
> > > Redirection of emails originated by PR actions to notifications@ seems
> > to
> > > be almost finished, so email flood issue seems to be already solved for
> > dev@
> > > .
> > >
> > > It is up to us to decide if we would like to migrate sooner. Later all
> > > repositories will be migrated anyway.
> > >
> > > Affected repositories
> > > - Ignite Release
> > > - Ignite
> > >
> > > Please share your opinion.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > -- Forwarded message -
> > > From: Daniel Gruno 
> > > Date: пт, 7 дек. 2018 г. в 19:52
> > > Subject: [NOTICE] Mandatory relocation of Apache git repositories on
> > > git-wip-us.apache.org
> > > To: us...@infra.apache.org 
> > >
> > >
> > > [IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
> > >   DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]
> > >
> > > Hello Apache projects,
> > >
> > > I am writing to you because you may have git repositories on the
> > > git-wip-us server, which is slated to be decommissioned in the coming
> > > months. All repositories will be moved to the new gitbox service which
> > > includes direct write access on github as well as the standard ASF
> > > commit access via gitbox.apache.org.
> > >
> > > ## Why this move? ##
> > > The move comes as a result of retiring the git-wip service, as the
> > > hardware it runs on is longing for retirement. In lieu of this, we
> > > have decided to consolidate the two services (git-wip and gitbox), to
> > > ease the management of our repository systems and future-proof the
> > > underlying hardware. The move is fully automated, and ideally, nothing
> > > will change in your workflow other than added features and access to
> > > GitHub.
> > >
> > > ## Timeframe for relocation ##
> > > Initially, we are asking that projects voluntarily request to move
> > > their repositories to gitbox, hence this email. The voluntary
> > > timeframe is between now and January 9th 2019, during which projects
> > > are free to either move over to gitbox or stay put on git-wip. After
> > > this phase, we will be requiring the remaining projects to move within
> > > one month, after which we will move the remaining projects over.
> > >
> > > To have your project moved in this initial phase, you will need:
> > >
> > > - Consensus in the project (documented via the mailing list)
> > > - File a JIRA ticket with INFRA to voluntarily move your project repos
> > >over to gitbox (as stated, this is highly automated and will take
> > >between a minute and an hour, depending on the size and number of
> > >your repositories)
> > >
> > > To sum up the preliminary timeline;
> > >
> > > - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
> > >relocation
> > > - January 9th -> February 6th: Mandated (coordinated) relocation
> > > - February 7th: All remaining repositories are mass migrated.
> > >
> > > This timeline may change to accommodate various scenarios.
> > >
> > > ## Using GitHub with ASF repositories ##
> > > When your project has moved, you are free to use either the ASF
> > > repository system (gitbox.apache.org) OR GitHub for your development
> > > and code pushes. To be able to use GitHub, please follow the primer
> > > at: https://reference.apache.org/committer/github
> > >
> > >
> > > We appreciate your understanding of this issue, and hope that your
> > > project can coordinate voluntarily moving your repositories in a
> > > timely manner.
> > >
> > > All settings, such 

Re: [ML][DISCUSSION] ML Package reorganization

[aplatonov]

Yes, I totally agree with you. But I think we should move "composition"
package to "algorithm"  because of classes of "composition" are
meta-algothms. Class ModelComposition with "aggregator"-package can be moved
to "model". Also, we should refactor Dataset-classes. There are two
different Dataset.java files at least. 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: newbie to Apache Ignite.

[Nikolay Izhikov]

Hello, Lawrence.


Can you share code to reproduce the issue?

чт, 13 дек. 2018 г., 0:15 lawrence@barclays.com.invalid:

> Hi,
>
>I just started looking to adapt Apache Ignite into our project as an
> in-memory distributed cache and potentially as distributed computing grid.
>
>An Ignite client application, ClientPutGetExample, encountered “Ignite
> cluster is unavailable” error intermittently.Ignite 2.7.0 was being
>
>used in the dev environment.There was  a cluster of 2 Ignite
> servers which was running with the client application in the same host.
>
>I noticed that the error was encountered after one of the server was
> stopped.
>
>
>
>How would I resolve the issue?
>
>Thank you for your help in advance.
>
>
>
> Regards,
>
> Lawrence
>
>
>
>
>
>
>
>
>
>
> _
>
> This message is for information purposes only, it is not a recommendation,
> advice, offer or solicitation to buy or sell a product or service nor an
> official confirmation of any transaction. It is directed at persons who are
> professionals and is not intended for retail customer use. Intended for
> recipient only. This message is subject to the terms at:
> www.barclays.com/emaildisclaimer.
>
> For important disclosures, please see:
> www.barclays.com/salesandtradingdisclaimer regarding market commentary
> from Barclays Sales and/or Trading, who are active market participants; and
> in respect of Barclays Research, including disclosures relating to specific
> issuers, please see http://publicresearch.barclays.com.
>
>
> __
> If you are incorporated or operating in Australia, please see
> https://www.home.barclays/disclosures/importantapacdisclosures.html for
> important disclosure.
>
> __
>
> __
> How we use personal information  see our privacy notice
> https://www.investmentbank.barclays.com/disclosures/personalinformationuse.html
>
>
> _
>

[MTCGA]: new failures in builds [2372233] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New stable failure of a flaky test in master 
GridServiceProcessorBatchDeploySelfTest.testDeployAllTopologyChangeFail 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5756858704963167219=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 04:10:57 13-12-2018 

[MTCGA]: new failures in builds [2364053] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New stable failure of a flaky test in master 
CacheAbstractTest.TestCacheConfigurationExpiryPolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7892614422846595039=%3Cdefault%3E=testDetails
 Changes may lead to failure were done by 
 - aplatonovv 
https://ci.ignite.apache.org/viewModification.html?modId=839641

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 07:55:53 13-12-2018 

Re: About readme.io's latest docs

[李玉珏]

Hi,

Does anyone in the community can help to export all the latest documents 
for me?


thanks very much.

在 2018/7/10 下午3:26, Dmitry Pavlov 写道:

Great, thanks for letting me know.

вт, 10 июл. 2018 г. в 4:01, 李玉珏@163 <18624049...@163.com 
>:


I have received the attachments.

在 2018/7/10 上午6:23, Prachi Garg 写道:
> I already sent him the export, but the attachments were huge. I
think
> that's why the email reply was not received by the dev list.
>
> -P
>
> On Mon, Jul 9, 2018 at 2:33 PM, Dmitry Pavlov
mailto:dpavlov@gmail.com>> wrote:
>
>> Igniters, was this question solved? Is it possible to do such
export?
>>
>> пт, 1 июн. 2018 г. в 19:40, 李玉珏@163 <18624049...@163.com
>:
>>
>>> Prachi,
>>>
>>>
>>> Can you help to export all the latest version of all documents
to me?
>>> I am ready to make a synchronization of the chinese version of the
>>> document.
>>>
>>> thanks!
>>>
>>>
>>> 在 2017/11/24 下午1:33, Prachi Garg 写道:
 Attached.

 On Wed, Nov 15, 2017 at 4:41 AM, 李玉珏@163 <18624049...@163.com

 >> wrote:

      Prachi,


      Can you help me to export all the latest version of all
documents
      to me?
      I am ready to make a synchronization of the chinese
version of the
      document.

      thanks!

      在 2017/8/17 上午2:05, Prachi Garg 写道:
>      Please see attached.
>
>      -P
>
>      On Wed, Aug 9, 2017 at 11:19 AM, Denis Magda
mailto:dma...@gridgain.com>
>      >> wrote:
>
>          Hi!
>
>          Prachi is on vacation and will send you the latest
version as
>          soon as she is back to work.
>
>          —
>          Denis
>
>>          On Aug 7, 2017, at 5:33 AM, 李玉珏@163
<18624049...@163.com 
>>          >> wrote:
>>
>>
>>          Prachi,
>>
>>
>>          Can you help me to export all the latest version
of all
>>          documents to me?
>>          I am ready to make a synchronization of the
chinese version
>>          of the document.
>>
>>          thanks!
>>
>>
>>          在 2017/5/13 上午12:48, Prachi Garg 写道:
>>>          See attached for cpp, .net and integrations
documentation.
>>>
>>>          -P
>>>
>>>          On Fri, May 12, 2017 at 9:47 AM, Prachi Garg
>>>          mailto:pg...@gridgain.com>
>> wrote:
>>>
>>>              See attached for java, file-system, and tools
>>>              documentation.
>>>
>>>              -P
>>>
>>>              On Fri, May 12, 2017 at 5:39 AM, 李玉珏@163
>>>              <18624049...@163.com
 >>
>>> wrote:
>>>                  Prachi,
>>>
>>>
>>>                  Can you help me to export all the latest
version of
>>>                  all documents to me?
>>>                  I am ready to make a synchronization of
the chinese
>>>                  version of the document.
>>>
>>>                  thanks!
>>>
>>>
>>>                  在 2017/3/18 04:54, Prachi Garg 写道:
                  Hi,

                  Please see attached.

                  -P

                  On Fri, Mar 17, 2017 at 4:56 AM, 李玉珏@163
                  <18624049...@163.com
 >>
                  wrote:

                      Prachi,


                      Can you help me to export all the latest
                      version of all documents to me?
                      I am ready to make a synchronization
of the
                      chinese version of the document.


                      在 2016/12/21 00:55, Prachi Garg 写道:
>                      Hi,
>                      I have attached the documentation for
> Hadoop/Spark and 

Re: CONTRIBUTING.md first version

[Павлухин Иван]

Dmitriy,

Thank you for your efforts! For me it looks exactly what is needed.
Screenshot will tell everything for me (see hint at right) [1].

[1] 
https://gist.githubusercontent.com/pavlukhin/c8c7c6266eeab56048c31f5cdfb31d20/raw/a9af17d920a6eda4328c301023229c2ba6332d58/contr.png
чт, 13 дек. 2018 г. в 01:23, Dmitriy Pavlov :
>
> There is nothing wrong with wiki (excepting it is sometimes way too slow
> and it may be annoying ).
>
> Github markdown is a way to
> 1. Keep documenting under version control system
> 2. Github recommends to provide this doc in the standard place.
>
> Of course, we can move this new doc content to wiki and just provide link
> from contributing.md to wiki. Should we?
>
> I can also add a copy to a wiki. Denis, folks,  please share your vision.
>
> чт, 13 дек. 2018 г., 0:32 Denis Magda :
>
> > What is wrong with Ignite wiki? It seems like the right place for the doc
> > like this.
> >
> > --
> > Denis
> >
> > On Wed, Dec 12, 2018 at 9:21 AM Dmitriy Pavlov  wrote:
> >
> > > Hi Igniters,
> > >
> > > Just to inform, the very first version of the Contribution step by step
> > > guide was added to GitHub. It is one more entry point for newcomers to
> > find
> > > out what and how should be done to contribute:
> > >
> > >
> > https://github.com/apache/ignite/blob/master/CONTRIBUTING.md#contributing-to-apache-ignite
> > >
> > >
> > > Feel free to share with newcomers and contribute.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> >



-- 
Best regards,
Ivan Pavlukhin

[jira] [Created] (IGNITE-10667) Web console: 'key store file' field is uneditable in SSL configuration

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-10667:
---

 Summary: Web console: 'key store file' field is uneditable in SSL 
configuration
 Key: IGNITE-10667
 URL: https://issues.apache.org/jira/browse/IGNITE-10667
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Konstantinov
Assignee: Vasiliy Sisko
 Attachments: image-2018-12-13-13-23-15-844.png

 !image-2018-12-13-13-23-15-844.png! 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: CONTRIBUTING.md first version

[Павлухин Иван]

(the link "contributing guidelines" on the screenshot refers to CONTRIBUTING.md)
чт, 13 дек. 2018 г. в 08:50, Павлухин Иван :
>
> Dmitriy,
>
> Thank you for your efforts! For me it looks exactly what is needed.
> Screenshot will tell everything for me (see hint at right) [1].
>
> [1] 
> https://gist.githubusercontent.com/pavlukhin/c8c7c6266eeab56048c31f5cdfb31d20/raw/a9af17d920a6eda4328c301023229c2ba6332d58/contr.png
> чт, 13 дек. 2018 г. в 01:23, Dmitriy Pavlov :
> >
> > There is nothing wrong with wiki (excepting it is sometimes way too slow
> > and it may be annoying ).
> >
> > Github markdown is a way to
> > 1. Keep documenting under version control system
> > 2. Github recommends to provide this doc in the standard place.
> >
> > Of course, we can move this new doc content to wiki and just provide link
> > from contributing.md to wiki. Should we?
> >
> > I can also add a copy to a wiki. Denis, folks,  please share your vision.
> >
> > чт, 13 дек. 2018 г., 0:32 Denis Magda :
> >
> > > What is wrong with Ignite wiki? It seems like the right place for the doc
> > > like this.
> > >
> > > --
> > > Denis
> > >
> > > On Wed, Dec 12, 2018 at 9:21 AM Dmitriy Pavlov  wrote:
> > >
> > > > Hi Igniters,
> > > >
> > > > Just to inform, the very first version of the Contribution step by step
> > > > guide was added to GitHub. It is one more entry point for newcomers to
> > > find
> > > > out what and how should be done to contribute:
> > > >
> > > >
> > > https://github.com/apache/ignite/blob/master/CONTRIBUTING.md#contributing-to-apache-ignite
> > > >
> > > >
> > > > Feel free to share with newcomers and contribute.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin



-- 
Best regards,
Ivan Pavlukhin

Re: Is it time to move forward to JUnit4 (5)?

[Павлухин Иван]

Hi Oleg,

Thanks your for detailed answer. Using junit4 annotations where
possible sounds good to me.

Also I would like to thank everybody involved in migration to junit4
process for your efforts. You were able to move a problem which looked
mountain-heavy. I believe that is a significant milestone on Ignite
stabilization road.

Well done!
вт, 11 дек. 2018 г. в 20:20, oignatenko :
>
> Hi Ivan,
>
> That's a very good question and I think you spotted a point where Ignite
> test developer's preferences need to change when migrating from Junit 3 to
> 4.
>
> In brief, when developing under JUnit 3 there were good reasons to prefer
> our homebrew template methods (beforeTestsStarted, beforeTest, afterTest,
> afterTestsStopped) over those provided by JUnit (setUp / tearDown). To
> answer your other question, in order to reliably understand how our methods
> work one would better go to (potentially changing) code of GridAbstractTest
> and check where and how these are invoked.
>
> But under JUnit 4 these reasons don't apply anymore and preference should go
> the other way, namely in favor of annotations @Before, @After, @BeforeClass,
> @AfterClass.
>
> In other words, when these annotations allow you implement desired test
> logic you better use them. These annotations have stable, well documented
> and widely understood semantics (and don't require one to skim through 3K
> lines of code for grasping details:).
>
> And it makes sense to "fallback" to our old-fashioned methods only when you
> simply have no other choice. For example doing some quick and simple fix in
> a legacy code that is filled with multiple usages of our old template
> methods it may be okay to choose to keep old style - instead of doing
> cumbersome, risky and lengthy refactoring or polluting test code with
> inconsistent use of annotations intertwined with old style methods.
>
> That's what I think. If you're interested in why I think so, refer to boring
> explanation in PS.
>
> regards, Oleg
>
> -
>
> PS. With regards to old preferences I believe it is very important to keep
> in mind that Junit 3 didn't provide means to do initialization and cleanup
> once for all test methods in the class.
>
> So when one needed things to run once per test class they had to go
> somewhere else. In Ignite the natural way to go was GridAbstractTest which
> provided this functionality via beforeTestsStarted and afterTestsStopped
> (with some glitches as I recently learned but still). And after one had to
> use these methods there was no compelling reason to use JUnit's setUp and
> tearDown, because GridAbstractTest also provided similar methods.
>
> It was just easier to consistently use full and cohesive set of methods
> provided by GridAbstractTest than switch back and forth between it and JUnit
> 3 methods.
>
> With JUnit 4 above reasoning doesn't apply anymore because its features are
> just as full and cohesive and can be consistently used without having to
> fallback to stuff from GridAbstractTest. So we have a real choice now and
> can judge based on other criteria than functional completeness (since both
> ways now satisfy that most important criteria).
>
> And this reveals us a full spectrum of JUnit advantages over
> GridAbstractTest (which was obscured before, because JUnit 3 was
> functionally incomplete).
>
> And on that newly opened spectrum JUnit 4 API wins hands down.
>
> JUnit 4 is much better documented and "googlable", it is stable and much
> wider known. The last but not the least, as I already mentioned, it doesn't
> require one to skim through 3K lines of code for grasping details and its
> semantics doesn't depend on (possibly changing) implementation code.
>
> Also, initialization and cleanup methods of JUnit 4 make an organic part of
> wider, more comprehensive API which smoothly integrates with IDE, Maven,
> Teamcity and all imaginable Java tools over there.
>
> Every time one annotates @Test or @Ignore gives them a reason to prefer
> other JUnit annotations over something else. It works about the same as I
> described above for beforeTestsStarted in GridAbstractTest and how it
> inhibited use of JUnit 3 methods: after you picked one part of some API it
> naturally becomes more convenient to keep using other parts of that API.
>
> Summing up, I believe that every time you have a (sensible) choice between
> JUnit 4 and GridAbstractTest methods you just pick JUnit 4 and don't look
> back.
>
>
> Павлухин Иван wrote
> > Ed, Oleg,
> >
> > Could you please clarify on the following point:
> >> 3. Add @Before, @After on methods which should run before, after a
> >> test (setUp, tearDown in current approach).
> > Beforehand we used to override beforeTestsStarted, beforeTest,
> > afterTest, afterTestsStarted. What will happen with these methods
> > after migration to junit4? I can see that order of execution can
> > become tricky especially when both styles are used at the same time.
> > And an immediate suggestion is to avoid using both 

Re: [jira] [Created] (IGNITE-10577) ignite-kubernetes is missing jackson-annotations dependency

[Stephen Darlington]

Yeah, you can fix by manually adding the dependency either in your own code or 
by adding ignite-aws or ignite-rest-http to your list of OPTION_LIBS (since 
those also contain the correct dependency).

Regards,
Stephen

> On 7 Dec 2018, at 14:45, Dmitriy Pavlov  wrote:
> 
> I've checked this patch and it looks good to me, it is really simple.
> 
> But probably we should consider auto-test it somehow. Nikolay Kulagin,
> could you chime in?
> 
> Stephen, could it be a workaround for end-user code? I guess user can just
> add this dependency in the pom and everything will work well.
> 
> Because release 2.7 is already done, the nearest time the issue could be
> fixed is 2.8.
> 
> чт, 6 дек. 2018 г. в 20:43, Stephen Darlington <
> stephen.darling...@gridgain.com>:
> 
>> Not sure what the etiquette on this is, but this is a regression in 2.7. I
>> added a simple patch to the ticket.
>> 
>> Regards,
>> Stephen
>> 
>>> On 6 Dec 2018, at 14:42, Stephen Darlington (JIRA) 
>> wrote:
>>> 
>>> Stephen Darlington created IGNITE-10577:
>>> ---
>>> 
>>>Summary: ignite-kubernetes is missing jackson-annotations
>> dependency
>>>Key: IGNITE-10577
>>>URL: https://issues.apache.org/jira/browse/IGNITE-10577
>>>Project: Ignite
>>> Issue Type: Bug
>>> Components: build
>>>   Affects Versions: 2.7
>>>   Reporter: Stephen Darlington
>>>   Assignee: Stephen Darlington
>>> 
>>> 
>>> When starting 2.7 with the ignite-kubernetes option I get the following
>> exception on startup:
>>> 
>>> {{[13:44:34,605][SEVERE][main][IgniteKernal] Got exception while
>> starting (will rollback startup
>> routine).}}{{java.lang.NoClassDefFoundError:
>> com/fasterxml/jackson/annotation/JsonView}}{{ at
>> com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.(JacksonAnnotationIntrospector.java:37)}}{{
>> at
>> com.fasterxml.jackson.databind.ObjectMapper.(ObjectMapper.java:291)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:151)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049)}}{{
>> at
>> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910)}}
>>> 
>>> It's clearly missing a dependency.
>>> 
>>> 
>>> 
>>> --
>>> This message was sent by Atlassian JIRA
>>> (v7.6.3#76005)
>> 
>> 
>> 

[jira] [Created] (IGNITE-10650) Yardstick pre-loader does not use defaultDataRegionConfiguration

[Ivan Artukhov (JIRA)]

Ivan Artukhov created IGNITE-10650:
--

 Summary: Yardstick pre-loader does not use 
defaultDataRegionConfiguration
 Key: IGNITE-10650
 URL: https://issues.apache.org/jira/browse/IGNITE-10650
 Project: Ignite
  Issue Type: Bug
  Components: yardstick
Affects Versions: 2.7
Reporter: Ivan Artukhov
 Fix For: 2.8


the new Loader.java class fails to get DataRegionConfiguration if it is not 
explicitly set for a given cache. Why not to try to get data region 
configuration from defaultDataRegionConfiguration property in this case?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Pre-touch for Ignite off-heap memory

[Nikolay Izhikov]

Hello, Stanislav.

As far as I can see, we have controversal ticket based on previous dev-list 
discussion:

IGNITE-9113 - Allocate memory for a data region when first cache assigned to 
this region is created [1]

I planned to implement it soon.
Looks like we should have several options of memory(data regions) allocation:

- allocate all on startup (AFAIK this is how current implementation 
behave)
- allocate all on startup AND pre touch.
- allocate specific data region for first assignment.
- allocate specific data region for first assignment AND pre touch.

What do you think?

[1] https://issues.apache.org/jira/browse/IGNITE-9113

В Вт, 11/12/2018 в 19:39 +0300, Stanislav Lukyanov пишет:
> Igniters,
> 
> What is being suggested here is an Ignite off-heap’s version of Java’s 
> -XX:+AlwaysPreTouch.
> The latter is known to be used to guarantee that the committed memory is 
> backed by physical RAM.
> This ensures that
> a) JVM doesn’t have to do it during the actual work (avoiding overhead for 
> physical page allocation, possible contention with page cache, etc)
> b) JVM fails fast if the Xmx is greater than available RAM
> c) New processes will not be able to claim the memory JVM took for itself
> 
> Currently one can’t get the same benefits for Ignite because we use off-heap 
> as well as heap.
> So, we can implement a similar feature for Ignite – and make sure the users 
> can get all the memory pre-touch benefits if they want.
> Of course, it impacts startup time so we should enable it by a configuration 
> property (I’d suggest a system property for now).
> 
> Are there any objections to implementing this?
> 
> Thanks,
> Stan
> 
> From: Павлухин Иван
> Sent: 31 октября 2018 г. 12:50
> To: dev@ignite.apache.org
> Subject: Re: Pre-touch for Ignite off-heap memory
> 
> Hi,
> 
> I did an experiment described above. I created a patch with pre-touch [1]
> and started a JVM with an option -XX:+AlwaysPreTouch and configured
> Ignite with equal values for initial and max sizes for each data region.
> I did several runs. I observed JVM crash dumps [2], [3]. Also it is easy
> to observe JVM OOM-killed.
> 
> [1] https://github.com/apache/ignite/pull/5220
> [2]
> https://gist.github.com/pavlukhin/e5e6605e9b4366627ba8d1aab42f#file-hs_err_pid5763-log
> [3]
> https://gist.github.com/pavlukhin/e5e6605e9b4366627ba8d1aab42f#file-hs_err_pid6411-log
> 
> вт, 30 окт. 2018 г. в 9:19, Павлухин Иван :
> 
> > Hi guys,
> > 
> > I am not aware that it is possible to run JVM in "allocation-free" fashion.
> > If you know that it is possible please share it. As I know JVM allocates
> > memory out of garbage collectable area for internal purposes like JIT,
> > GC itself. Also native built-it code can request memory allocation from OS,
> > e.g. zip facilities. If we imagine that system is running under memory
> > allocation
> > which is close to a limit then even small allocation request can fail and
> > lead
> > to OOM killing.
> > 
> > But I think that a simple and useful thing that could be done first is
> > making
> > an experiment. As Andrey mentioned
> > > AFAIK, Ignite always pre-touch first region. So, you can try to set
> > 
> > region
> > > MAX size equal to MIN and get region allocated on node start.
> > 
> > If it is so then it should not be hard to launch Ignite and observe it
> > running
> > very close to OS memory limit with Java heap and Ignite off-heap
> > pre-touched.
> > After that one could check whether it is possible to observe Ignite OOM
> > killed.
> > Let's say my bet is that it is relatively easy to catch OOM here.
> > 
> > What do you think?
> > 
> > пт, 26 окт. 2018 г. в 18:18, Yakov Zhdanov :
> > 
> > > Andrey,
> > > 
> > > Probability of a OOM kill will be much lower if offheap is pretouched.
> > > What
> > > do you mean by JVM internal needs? In my understanding if user enables
> > > option to pretouch heap and fixes the heap to prevent jvm releasing memory
> > > back to OS, then OOM killing is very unlikely.
> > > 
> > > I would agree that pretouch for offheap may be helpful in many cases.
> > > 
> > > --Yakov
> > > 
> > 
> > 
> > --
> > Best regards,
> > Ivan Pavlukhin
> > 
> 
> 


signature.asc
Description: This is a digitally signed message part

[jira] [Created] (IGNITE-10649) Add documentation for control.sh about SSL

[Sergey Antonov (JIRA)]

Sergey Antonov created IGNITE-10649:
---

 Summary: Add documentation for control.sh about SSL
 Key: IGNITE-10649
 URL: https://issues.apache.org/jira/browse/IGNITE-10649
 Project: Ignite
  Issue Type: Task
  Components: documentation
Reporter: Sergey Antonov
 Fix For: 2.8






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSSION] Design document. Rebalance caches by transferring partition files

[Vladimir Ozerov]

Maxim,

Thank you for excellent analysis! From profiling data I see the following:
1) Almost no parallelism - one rebalance thread is used (default), two
responses are sent per a single demand request (default)
2) All system resources are underutilized - CPU, disk, network
3) Huge hotspot ion free lists

In general I would recommend to consider the following points during
further rebalance optimization:
1) Start with the fact that rebalance always causes system degradation due
to additional hardware resources required. Different deployments may
require different degradation modes. Sometimes "soft" mode is preferable -
long rebalance with low system overhead. This is what we see now. Sometimes
the opposite - as short rebalance as possible at the cost of severe
degradation in operations. Sometimes - something in the middle. Every
optimization we made should have clear explanation on how system degrades.
2) We need to investigate the hotspot on free lists. Looks like this is the
main limiting factor for now. Alex, do you have any ideas what is this? Is
it possible to bypass freelists completely during rebalance at the cost of
higher data fragmentation during concurrent updates?
3) We need to investigate streaming rebalance mode, when supplier
constantly streams data to demander similarly to our data streamer. It
should be fairly easy to implement, applicable for all modes and may
speedup rebalance up to 5-10 times. Great thing about this approach is that
it will allow users to choose between system stress level and rebalance
throughput easily.
4) File transfer rebalance: we need to have clear design of failure and
concurrency cases and degradation modes. Several questions to answer:
4.1) What would happen if another rebalance starts when previous is not
finished yet?
4.2) What would happen if supplier or demander fails in the middle? What
kind of cleanup would be required
4.3) Degradation: what kind of problems should users expect due to massive
disk and network load during file transfer and due to data merging on
demander side?
4.4) Degradation: how secondary indexes would be rebuilt on demander side?
Note that until indexes are ready node is not operational and cannot become
partition owner, and index rebuild is essentially full data rescan with
potentially the same issues with slow updates of persistent data structures
we have now.

Vladimir.

On Fri, Dec 7, 2018 at 3:32 PM Maxim Muzafarov  wrote:

> Vladimir,
>
>
> Let me propose to consider the whole this rebalance process as having
> three strategies:
> - The classical message-based approach, preferable to use for in-memory
> caches;
> - Historical rebalance based on WAL, used for rebalancing persisted
> caches deltas;
> - (new) File-based rebalance (current IEP-28), used for relocation of
> full cache partitions.
>
>
> First of all, I want to show you that for the full cache relocation
> file-based rebalancing strategy from my point has a set of advantages
> prior to the message-based approach. Let's also assume that the time
> spent on WAL logging during the rebalance procedure is already
> optimized (we are not taking it into account at all).
>
> According to preliminary measurements [8] and the message above we
> spend more than 65% of rebalancing time on creating K-V cache pair for
> preloading entries and supporting internal data structures. It is true
> as for in-memory cluster configuration and for a cluster with enabled
> persistence. It is also true, that these data structures can be used
> more efficiently by implementing batch entry processing for them. And
> it should be done (a ticket for it is already created [3]).
>
> Let's have a look closer to the simple example.
>
> I've collected some information about a cache on my stress-testing cluster:
> partitions (total): 65534
> single partition size: 437 MB
> rebalance batch: 512 Kb
> batches per partition: 874
> partitions per node: 606
> batches per node: 529644
>
> Let's assume that we've already implemented batched entry processing
> and we perform bulk operations over internal data structures.
> Regarding these assumptions, we still need to process 874 batches per
> each cache partition to transfer data. I will cost us up to 15 seconds
> per each partition file, a lot of CPU cycles to maintain internal data
> structures and block for a while other threads waiting for releasing
> database checkpoint lock.
>
> Increasing the rebalance batch size is not efficient here because we
> are starting to hold the database lock for too long. It will lead to
> thread starvation and will only slow down the whole rebalance speed.
> Exactly the same as increasing batch size, making the rebalance thread
> pool bigger can lead to the cluster performance drop for almost the
> same reasons.
>
> I think the file-based rebalance can provide us (prior to the batch
> processing) for huge caches:
>  - a fair non-blocking approach in each part of the rebalancing procedure;
>  - reduce the number of locks being 

Re: Schema in CacheConfig is not updated after DDL commands(Add/drop column, Create/drop index)

[Vladimir Ozerov]

Hi Nikolay,

I think yes.

On Mon, Dec 10, 2018 at 9:10 PM Nikolay Izhikov  wrote:

> Vladimir.
>
> Since we are talking about internal component(Spark Data Frame integration)
> can we use same technique as JdbcRequestHandler to obtain columns metadata?
> Seems, code above will fetch up to date columns info:
>
>
> org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler#getColumnsMeta
> ```
>
> for (String cacheName : ctx.cache().publicCacheNames()) {
> for (GridQueryTypeDescriptor table :
> ctx.query().types(cacheName)) {
> if (!matches(table.schemaName(), req.schemaName()))
> continue;
>
> if (!matches(table.tableName(), req.tableName()))
> continue;
> //Process column here
>
> ```
>
> пн, 19 нояб. 2018 г. в 18:54, Vladimir Ozerov :
>
> > There is no public API currently. This information can be extracted using
> > JDBC metadata requests.
> >
> > On Mon, Nov 19, 2018 at 6:50 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Vladimir,
> > >
> > > What is the best way to get current schema information (list of tables,
> > > columns, etc.)?
> > >
> > > -Val
> > >
> > > On Mon, Nov 19, 2018 at 1:21 AM Vladimir Ozerov 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > In this case Spark integration should be fixed. as we never stated
> that
> > > DDL
> > > > updates will be reflected in IgniteCache.getConfiguration().
> > > >
> > > >
> > > > On Mon, Nov 19, 2018 at 11:58 AM Ray  wrote:
> > > >
> > > > > When user performs column and index modification operation in
> SQL(ex
> > > > create
> > > > > index, drop index, add column, drop column),  QueryEntity in
> > > > > CacheConfiguration for the modified cache is not updated.
> > > > >
> > > > > Here's my analysis
> > > > >
> > > > > QueryEntity in QuerySchema is a local copy of the original
> > QueryEntity,
> > > > so
> > > > > the original QueryEntity is not updated when modification happens.
> > > > >
> > > > > I have created a ticket for this issue
> > > > > https://issues.apache.org/jira/browse/IGNITE-10314
> > > > >
> > > > > But as Vlad said in the comments "public configuration is
> immutable,
> > it
> > > > > represents initial cache parameters. So it is expected that
> > > configuration
> > > > > will not be updated after DDL commands. Real changes are
> accumulated
> > in
> > > > > separate query entity which is hidden from user and used
> internally"
> > > > >
> > > > > But I think it's only reasonable to return the newest QueryEntity
> to
> > > > user.
> > > > >
> > > > > For example, a user adds a column to a table then he reads data
> using
> > > > Spark
> > > > > data frame API which currently relies on QueryEntity to construct
> > data
> > > > > frame
> > > > > schema, so user will get wrong schema.
> > > > >
> > > > > What do you guys think?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (IGNITE-10651) GridCachePartitionExchangeManager.affinityReadyFuture() should return GridDhtTopologyFuture instead of IgniteInternalFuture

[Vyacheslav Koptilin (JIRA)]

Vyacheslav Koptilin created IGNITE-10651:


 Summary: GridCachePartitionExchangeManager.affinityReadyFuture() 
should return GridDhtTopologyFuture instead of IgniteInternalFuture
 Key: IGNITE-10651
 URL: https://issues.apache.org/jira/browse/IGNITE-10651
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.7
Reporter: Vyacheslav Koptilin


Under some circumstances, 
{{GridCachePartitionExchangeManager.affinityReadyFuture()}}returns 
{{GridFinishedFuture}} or 
{{GridCachePartitionExchangeManager.AffinityReadyFuture}}
Obviously, the returned future cannot be used in order to validate a cache. It 
seems that this behavior can be improved if {{affinityReadyFuture()}} returns 
an instance of {{GridDhtTopologyFuture,}} and therefore, it will simplify the 
logic related to cache validation.

Perhaps, {{ExchangeFutureSet}} can be changed to {{ExchangeFutureMap}} that 
should contain a mapping of topology version to the corresponding 
{{GridDhtTopologyFuture}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10652) LocalWalModeNoChangeDuringRebalanceOnNonNodeAssignTest fails on TC

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-10652:
-

 Summary: LocalWalModeNoChangeDuringRebalanceOnNonNodeAssignTest 
fails on TC
 Key: IGNITE-10652
 URL: https://issues.apache.org/jira/browse/IGNITE-10652
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Reporter: Andrew Mashenkov
 Fix For: 2.8


LocalWalModeNoChangeDuringRebalanceOnNonNodeAssignTest fails with TRANSACTIONAL 
cache.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ML][DISCUSSION] ML Package reorganization

[Alexey Zinoviev]

Also, I've put my 5 cents about approach of moving files. Please do it with
patience to avoid erasion of git history.

The ugly future: @ybabak is author of all ml files and we couldnt find real
author of class.

The well future: moving files without +- many rows

ср, 12 дек. 2018 г., 15:10 Dmitriy Pavlov dpav...@apache.org:

> Hi Yuri,
>
> Would it affect users?
>
> Ignite-code has separation of API classes (not internal) and internal
> stuff. Internal classes may be changed from version to version, but API
> can't be renamed/moved/and so on. What about ML?
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 11 дек. 2018 г. в 16:56, Yuriy Babak :
>
> > Igniters,
> >
> > I want to discuss package structure for ML module. Actually I don't like
> > our current package organization. The main problem is that we have lots
> of
> > different packages under root package.
> >
> > For examples algorithm related packages on same level like some utility
> > infrastructure packages such as environment(package with some classes
> > related with training environment).
> >
> > So I created branch [1] with example of new package structure. Please
> share
> > any feedback about this package structure and about this idea itself.
> >
> > Regards,
> > Yuriy.
> >
> > [1] -
> >
> >
> https://github.com/YuriBabak/ignite/tree/ignite-10641/modules/ml/src/main/java/org/apache/ignite/ml
> >
>

Re: [ML][DISCUSSION] ML Package reorganization

[Alexey Zinoviev]

I agree, that we should worry about users, but according User List we dont
have significant number of ml users. I think that next 1-2 releases we
could make changes in api and package structure



ср, 12 дек. 2018 г., 15:10 Dmitriy Pavlov dpav...@apache.org:

> Hi Yuri,
>
> Would it affect users?
>
> Ignite-code has separation of API classes (not internal) and internal
> stuff. Internal classes may be changed from version to version, but API
> can't be renamed/moved/and so on. What about ML?
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 11 дек. 2018 г. в 16:56, Yuriy Babak :
>
> > Igniters,
> >
> > I want to discuss package structure for ML module. Actually I don't like
> > our current package organization. The main problem is that we have lots
> of
> > different packages under root package.
> >
> > For examples algorithm related packages on same level like some utility
> > infrastructure packages such as environment(package with some classes
> > related with training environment).
> >
> > So I created branch [1] with example of new package structure. Please
> share
> > any feedback about this package structure and about this idea itself.
> >
> > Regards,
> > Yuriy.
> >
> > [1] -
> >
> >
> https://github.com/YuriBabak/ignite/tree/ignite-10641/modules/ml/src/main/java/org/apache/ignite/ml
> >
>

Re: [ML][DISCUSSION] ML Package reorganization

[Dmitriy Pavlov]

Hi Yuri,

Would it affect users?

Ignite-code has separation of API classes (not internal) and internal
stuff. Internal classes may be changed from version to version, but API
can't be renamed/moved/and so on. What about ML?

Sincerely,
Dmitriy Pavlov

вт, 11 дек. 2018 г. в 16:56, Yuriy Babak :

> Igniters,
>
> I want to discuss package structure for ML module. Actually I don't like
> our current package organization. The main problem is that we have lots of
> different packages under root package.
>
> For examples algorithm related packages on same level like some utility
> infrastructure packages such as environment(package with some classes
> related with training environment).
>
> So I created branch [1] with example of new package structure. Please share
> any feedback about this package structure and about this idea itself.
>
> Regards,
> Yuriy.
>
> [1] -
>
> https://github.com/YuriBabak/ignite/tree/ignite-10641/modules/ml/src/main/java/org/apache/ignite/ml
>

Re: [ML][DISCUSSION] ML Package reorganization

[aplatonov]

Yes, I totally agree with you. But I think we should move "composition"
package to "algorithm"  because of classes of "composition" are
meta-algothms. Class ModelComposition with "aggregator"-package can be moved
to "model". Also, we should refactor Dataset-classes. There are two
different Dataset.java files at least. 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [ML][DISCUSSION] ML Package reorganization

[Yuriy Babak]

Dmitry,

Yes, those changes will affect users. But we could postpone this change to
the next major release(3.0) and we could implement a simple migration tool
for the ML part.

Regards,
Yuriy

ср, 12 дек. 2018 г. в 15:10, Dmitriy Pavlov :

> Hi Yuri,
>
> Would it affect users?
>
> Ignite-code has separation of API classes (not internal) and internal
> stuff. Internal classes may be changed from version to version, but API
> can't be renamed/moved/and so on. What about ML?
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 11 дек. 2018 г. в 16:56, Yuriy Babak :
>
> > Igniters,
> >
> > I want to discuss package structure for ML module. Actually I don't like
> > our current package organization. The main problem is that we have lots
> of
> > different packages under root package.
> >
> > For examples algorithm related packages on same level like some utility
> > infrastructure packages such as environment(package with some classes
> > related with training environment).
> >
> > So I created branch [1] with example of new package structure. Please
> share
> > any feedback about this package structure and about this idea itself.
> >
> > Regards,
> > Yuriy.
> >
> > [1] -
> >
> >
> https://github.com/YuriBabak/ignite/tree/ignite-10641/modules/ml/src/main/java/org/apache/ignite/ml
> >
>

[jira] [Created] (IGNITE-10653) [ML] Make basic blocks of function combinations contain toString description

[Artem Malykh (JIRA)]

Artem Malykh created IGNITE-10653:
-

 Summary: [ML] Make basic blocks of function combinations contain 
toString description
 Key: IGNITE-10653
 URL: https://issues.apache.org/jira/browse/IGNITE-10653
 Project: Ignite
  Issue Type: Wish
Reporter: Artem Malykh


It may be useful to have description for identity, constant, combine and maybe 
other parts of combination chain have meaningful descriptions to make debugging 
less painful.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[MTCGA]: new failures in builds [2527852] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *Recently contributed test failed in master 
ClientSslParametersTest.testNonExistentCipherSuite 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2251849292403456778=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
ClientSslParametersTest.testNonExistentProtocol 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7837765355690376078=%3Cdefault%3E=testDetails
 Changes may lead to failure were done by 
 - akuznetsov 
https://ci.ignite.apache.org/viewModification.html?modId=844317

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 17:10:54 12-12-2018 

[jira] [Created] (IGNITE-10658) Rebalance status in Visor stays on 99.99%.

[Pavel Voronkin (JIRA)]

Pavel Voronkin created IGNITE-10658:
---

 Summary: Rebalance status in Visor stays on 99.99%.
 Key: IGNITE-10658
 URL: https://issues.apache.org/jira/browse/IGNITE-10658
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Voronkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10659) Possible deadlock causing by metadata request in grid-timeout-worker

[Sergey Kosarev (JIRA)]

Sergey Kosarev created IGNITE-10659:
---

 Summary: Possible deadlock causing by metadata request  in 
grid-timeout-worker
 Key: IGNITE-10659
 URL: https://issues.apache.org/jira/browse/IGNITE-10659
 Project: Ignite
  Issue Type: Bug
Reporter: Sergey Kosarev


It looks like IGNITE-9840 fixes not all the cases.
We have similar problem on a sever node:

{code}
Thread [name="grid-timeout-worker-#119%DPL_GRID%DplGridNodeName%", id=235, 
state=WAITING, blockCnt=2, waitCnt=664073]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
at 
o.a.i.i.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata0(CacheObjectBinaryProcessorImpl.java:592)
at 
o.a.i.i.processors.cache.binary.CacheObjectBinaryProcessorImpl.metadata(CacheObjectBinaryProcessorImpl.java:550)
at 
o.a.i.i.processors.cache.binary.CacheObjectBinaryProcessorImpl$1.metadata(CacheObjectBinaryProcessorImpl.java:200)
at o.a.i.i.binary.BinaryContext.metadata(BinaryContext.java:1266)
at o.a.i.i.binary.BinaryUtils.type(BinaryUtils.java:2425)
at o.a.i.i.binary.BinaryObjectImpl.rawType(BinaryObjectImpl.java:302)
at 
o.a.i.i.binary.BinaryObjectExImpl.toString(BinaryObjectExImpl.java:208)
at 
o.a.i.i.binary.BinaryObjectExImpl.appendValue(BinaryObjectExImpl.java:286)
at 
o.a.i.i.binary.BinaryObjectExImpl.toString(BinaryObjectExImpl.java:235)
at 
o.a.i.i.binary.BinaryObjectExImpl.toString(BinaryObjectExImpl.java:187)
at o.a.i.i.binary.BinaryObjectImpl.toString(BinaryObjectImpl.java:920)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at 
o.a.i.i.processors.cache.transactions.TxEntryValueHolder.toString(TxEntryValueHolder.java:161)
at java.lang.String.valueOf(String.java:2994)
at o.a.i.i.util.GridStringBuilder.a(GridStringBuilder.java:101)
at o.a.i.i.util.tostring.SBLimitedLength.a(SBLimitedLength.java:100)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:849)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1067)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:994)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:754)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:722)
at 
o.a.i.i.processors.cache.transactions.IgniteTxEntry.toString(IgniteTxEntry.java:1273)
at java.lang.String.valueOf(String.java:2994)
at o.a.i.i.util.GridStringBuilder.a(GridStringBuilder.java:101)
at o.a.i.i.util.tostring.SBLimitedLength.a(SBLimitedLength.java:100)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:849)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:807)
at 
o.a.i.i.util.tostring.GridToStringBuilder.addCollection(GridToStringBuilder.java:900)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:845)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:807)
at 
o.a.i.i.util.tostring.GridToStringBuilder.appendVals(GridToStringBuilder.java:1662)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1070)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:994)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:754)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:722)
at 
o.a.i.i.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:491)
at java.lang.String.valueOf(String.java:2994)
at o.a.i.i.util.GridStringBuilder.a(GridStringBuilder.java:101)
at o.a.i.i.util.tostring.SBLimitedLength.a(SBLimitedLength.java:100)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:849)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toStringImpl0(GridToStringBuilder.java:1067)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:994)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:703)
at 
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:662)
at 
o.a.i.i.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:1621)
at 

[jira] [Created] (IGNITE-10660) Can't drop index created for dynamic cache

[Alex Volkov (JIRA)]

Alex Volkov created IGNITE-10660:


 Summary: Can't drop index created for dynamic cache
 Key: IGNITE-10660
 URL: https://issues.apache.org/jira/browse/IGNITE-10660
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.7
Reporter: Alex Volkov


Here are steps to reproduce:
 # Start cluster with persistence.
 # Start cache with configuration (transactional, replicated full_sync with 
query entity).
 # Connect to cluster using sqlline.
 # Create index on this cache.
 # Try to drop index created on step 4.

Expecting result:

Index dropped.

Actual result:

Got error:
{code:java}
[avolkov@lab37 bin]$ ./sqlline.sh --color=true --verbose=true 
--showWarnings=true --showNestedErrs=true --outputFormat=csv -u 
jdbc:ignite:thin://lab37:10800
Setting property: [showWarnings, true]
Setting property: [showNestedErrs, true]
Setting property: [outputFormat, csv]
issuing: !connect jdbc:ignite:thin://lab37:10800 '' '' 
org.apache.ignite.IgniteJdbcThinDriver
Connecting to jdbc:ignite:thin://lab37:10800
Connected to: Apache Ignite (version 2.7.1#20181210-sha1:a39dbf0f)
Driver: Apache Ignite Thin JDBC Driver (version 2.7.1#20181210-sha1:a39dbf0f)
Autocommit status: true
Transaction isolation: TRANSACTION_REPEATABLE_READ
sqlline version 1.3.0

0: jdbc:ignite:thin://lab37:10800> !tables
'TABLE_CAT','TABLE_SCHEM','TABLE_NAME','TABLE_TYPE','REMARKS','TYPE_CAT','TYPE_SCHEM','TYPE_NAME','SELF_REFERENCING_COL_NAME','REF_GENERATION'
'','cache_group_1_015','ALLTYPESINDEXED','TABLE','','','','','',''
'','cache_group_1_001','ALLTYPESINDEXED','TABLE','','','','','',''
'','cache1','ALLTYPESINDEXED','TABLE','','','','','',''

0: jdbc:ignite:thin://lab37:10800> select * from "cache1".ALLTYPESINDEXED;
'STRCOL','LONGCOL'
No rows selected (0.576 seconds)

0: jdbc:ignite:thin://lab37:10800> create index test_idx_1 on 
"cache1".ALLTYPESINDEXED(STRCOL);
No rows affected (0.05 seconds)

0: jdbc:ignite:thin://lab37:10800> drop index test_idx_1;
Error: Index doesn't exist: TEST_IDX_1 (state=42000,code=3006)
java.sql.SQLException: Index doesn't exist: TEST_IDX_1
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:750)
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:212)
at 
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:475)
at sqlline.Commands.execute(Commands.java:823)
at sqlline.Commands.sql(Commands.java:733)
at sqlline.SqlLine.dispatch(SqlLine.java:795)
at sqlline.SqlLine.begin(SqlLine.java:668)
at sqlline.SqlLine.start(SqlLine.java:373)
at sqlline.SqlLine.main(SqlLine.java:265)
0: jdbc:ignite:thin://lab37:10800> !index
'TABLE_CAT','TABLE_SCHEM','TABLE_NAME','NON_UNIQUE','INDEX_QUALIFIER','INDEX_NAME','TYPE','ORDINAL_POSITION','COLUMN_NAME','ASC_OR_DESC','CARDINALITY','PAGES','FILTER_CONDITION'
'','cache_group_1_001','ALLTYPESINDEXED','true','','ALLTYPESINDEXED_LONGCOL_IDX','3','0','LONGCOL','A','0','0',''
'','cache1','ALLTYPESINDEXED','true','','ALLTYPESINDEXED_STRCOL_ASC_IDX','3','0','STRCOL','A','0','0',''
'','cache1','ALLTYPESINDEXED','true','','TEST_IDX_1','3','0','STRCOL','A','0','0',''
'','cache_group_1_015','ALLTYPESINDEXED','true','','ALLTYPESINDEXED_LONGCOL_IDX','3','0','LONGCOL','A','0','0',''
{code}
 

In node log I see the exception:

 
{code:java}
[16:01:15,412][SEVERE][client-connector-#179][JdbcRequestHandler] Failed to 
execute SQL query [reqId=0, req=JdbcQueryExecuteRequest [schemaName=PUBLIC, 
pageSize=1024, maxRows=0, sqlQry=drop index test_idx_1, args=[], 
stmtType=ANY_STATEMENT_TYPE, autoCommit=true]]
class org.apache.ignite.internal.processors.query.IgniteSQLException: Index 
doesn't exist: TEST_IDX_1
at 
org.apache.ignite.internal.processors.query.h2.ddl.DdlStatementsProcessor.convert(DdlStatementsProcessor.java:644)
at 
org.apache.ignite.internal.processors.query.h2.ddl.DdlStatementsProcessor.runDdlStatement(DdlStatementsProcessor.java:244)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryDistributedSqlFieldsNative(IgniteH2Indexing.java:1977)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:2141)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2135)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2130)
at 
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2707)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2144)
at 
org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.executeQuery(JdbcRequestHandler.java:511)
at 

Re: Use of marshaller at node startup routine (need advice)

[Denis Mekhanikov]

Slava,

Interface *Service *extends *Serializable.* So, all services are supposed
to be serializable by the JdkMarshaller.
Usage of *BinaryMarshaller* or *OptimizedMarshaller* makes sense only from
performance point of view.
But I don't think, that we should try too hard to minimize the performance
impact of services serialization,
since it doesn't happen too often.

There are some tests like *IgniteServiceConfigVariationsFullApiTest*, that
check, that services, which are not
serializable, can be successfully deployed. I think, these tests should be
removed.
It's reasonable to require serializability, since all *Services* are marked
as *Serializable*, as I already mentioned.

Slava and I discussed a possibility to choose a marshaller, depending on
the node state.
If a node is already connected to the cluster, then it could use a binary
marshaller,
otherwise JDK marshaller could be used.
I think, if we decide to do so, then it will complicate logic and confuse
users.
This problem exists only for static services. They are not different from
dynamic ones,
except for the way of configuration, and a moment of deployment.
I don't see, why different constraints should be applied to them.

So, I'm for using the *JdkMarshaller* regardless of the service type or a
node state.

Denis

пт, 7 дек. 2018 г. в 15:57, Vyacheslav Daradur :

> Igniters, I need your advice about the following problem:
>
> It is necessary to serialize an object (just convert an object to
> bytes array) for including it in joining node data (DiscoveryDataBag)
> *at node startup routine*.
>
> The marshalling hangs If we use 'BinaryMarshaller' or
> 'OptimizedMarshaler' because class can't be registered in
> MarshallerContextImpl#registerClassName -> transport.proposeMapping on
> account of the request which can't be sent through discovery-spi at
> the moment.
>
> Also, 'JdkMarshaller' can't be used because it imposes limits on
> objects that should implement 'Serializable' interface. But this
> restriction is unacceptable for the case.
>
> As a workaround solution, an external library, like KRYO, can be used.
>
> What tools also available in the project to solve this problem?
>
> --
> Best Regards, Vyacheslav D.
>

[jira] [Created] (IGNITE-10654) Report in case of creating index with already existing fields collection.

[Stanilovsky Evgeny (JIRA)]

Stanilovsky Evgeny created IGNITE-10654:
---

 Summary: Report in case of creating index with already existing 
fields collection.
 Key: IGNITE-10654
 URL: https://issues.apache.org/jira/browse/IGNITE-10654
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.7
Reporter: Stanilovsky Evgeny
Assignee: Stanilovsky Evgeny
 Fix For: 2.8


Report in log if new index creating with already existing fields collection.

for example, need to log warn here:
{code:java}
cache.query(new SqlFieldsQuery("create index \"idx1\" on Val(keyStr, 
keyLong)"));
cache.query(new SqlFieldsQuery("create index \"idx3\" on Val(keyStr, 
keyLong)"));
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Use of marshaller at node startup routine (need advice)

[Vyacheslav Daradur]

Denis, thank you for the answer.

I've already implemented the same approach. It works well, new and old
tests have passed.
On Wed, Dec 12, 2018 at 3:47 PM Denis Mekhanikov  wrote:
>
> Slava,
>
> Interface *Service *extends *Serializable.* So, all services are supposed
> to be serializable by the JdkMarshaller.
> Usage of *BinaryMarshaller* or *OptimizedMarshaller* makes sense only from
> performance point of view.
> But I don't think, that we should try too hard to minimize the performance
> impact of services serialization,
> since it doesn't happen too often.
>
> There are some tests like *IgniteServiceConfigVariationsFullApiTest*, that
> check, that services, which are not
> serializable, can be successfully deployed. I think, these tests should be
> removed.
> It's reasonable to require serializability, since all *Services* are marked
> as *Serializable*, as I already mentioned.
>
> Slava and I discussed a possibility to choose a marshaller, depending on
> the node state.
> If a node is already connected to the cluster, then it could use a binary
> marshaller,
> otherwise JDK marshaller could be used.
> I think, if we decide to do so, then it will complicate logic and confuse
> users.
> This problem exists only for static services. They are not different from
> dynamic ones,
> except for the way of configuration, and a moment of deployment.
> I don't see, why different constraints should be applied to them.
>
> So, I'm for using the *JdkMarshaller* regardless of the service type or a
> node state.
>
> Denis
>
> пт, 7 дек. 2018 г. в 15:57, Vyacheslav Daradur :
>
> > Igniters, I need your advice about the following problem:
> >
> > It is necessary to serialize an object (just convert an object to
> > bytes array) for including it in joining node data (DiscoveryDataBag)
> > *at node startup routine*.
> >
> > The marshalling hangs If we use 'BinaryMarshaller' or
> > 'OptimizedMarshaler' because class can't be registered in
> > MarshallerContextImpl#registerClassName -> transport.proposeMapping on
> > account of the request which can't be sent through discovery-spi at
> > the moment.
> >
> > Also, 'JdkMarshaller' can't be used because it imposes limits on
> > objects that should implement 'Serializable' interface. But this
> > restriction is unacceptable for the case.
> >
> > As a workaround solution, an external library, like KRYO, can be used.
> >
> > What tools also available in the project to solve this problem?
> >
> > --
> > Best Regards, Vyacheslav D.
> >



-- 
Best Regards, Vyacheslav D.

[jira] [Created] (IGNITE-10655) .NET client should add peerClassLoading property in IgniteConfiguration class

[Andrey Aleksandrov (JIRA)]

Andrey Aleksandrov created IGNITE-10655:
---

 Summary: .NET client should add peerClassLoading property in 
IgniteConfiguration class
 Key: IGNITE-10655
 URL: https://issues.apache.org/jira/browse/IGNITE-10655
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.7
Reporter: Andrey Aleksandrov
 Fix For: 2.8


In case if java server nodes in a cluster were set up with peerClassLoading set 
to true then there is no way to start node via .NET client using 
IgniteConfiguration (because the default value is false).

The only way to do it is using of the XML configuration.

Possibility to set this property from .Net client programmatically should be 
added.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10656) SQL: Multiple columns could be used for partition pruning

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-10656:


 Summary: SQL: Multiple columns could be used for partition pruning
 Key: IGNITE-10656
 URL: https://issues.apache.org/jira/browse/IGNITE-10656
 Project: Ignite
  Issue Type: Task
Reporter: Vladimir Ozerov
 Fix For: 2.8






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10657) Thin clients (ODBC, JDBC, Java) should call ctx.security().onSessionExpired(subjid) on disconnect to avoid the possible leaking of the memory

[Andrey Aleksandrov (JIRA)]

Andrey Aleksandrov created IGNITE-10657:
---

 Summary: Thin clients (ODBC, JDBC, Java) should call 
ctx.security().onSessionExpired(subjid) on disconnect to avoid the possible 
leaking of the memory
 Key: IGNITE-10657
 URL: https://issues.apache.org/jira/browse/IGNITE-10657
 Project: Ignite
  Issue Type: Bug
  Components: thin client
Affects Versions: 2.7
Reporter: Andrey Aleksandrov
Assignee: Andrey Aleksandrov
 Fix For: 2.8


At the moment all classes that implement 
ClientListenerAbstractConnectionContext doesn't call the 
ctx.security().onSessionExpired(subjid) in disconect.

It provides the leaking of the memory related to resources that were generated 
during authentification.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Use of marshaller at node startup routine (need advice)

[Vyacheslav Daradur]

A bit more:

I implemented a solution which regards a node connection state: if the
node already joined to the cluster, then configured marshaller will be
used, otherwise, JdkMarshaller will be used.

This affects static configurations if a service can't be serialized
this will be logged (covered by test), it will be quite transparent
for users.

In general, I'm also for using JdkMarshaller regardless of conditions.

But, it would be not friendly for our users who already use
'Binarylizable' interface for the services.

I think 'IgniteServiceConfigVariationsFullApiTest' should not be
removed at the moment since we don't delete the previous
implementation. But we can fix it if needed.

As you said, we have 2 approaches:
1) Use `JdkMarshaller` since 'Service' interface extends
`Serializable` (and fix all the tests)
2) Choose the marshaller depending on the node's connection state;

Both approaches seem good to me, in the current PR second approach has
been implemented.

It would be great if other Ignite's experts said their opinion.

On Wed, Dec 12, 2018 at 3:52 PM Vyacheslav Daradur  wrote:
>
> Denis, thank you for the answer.
>
> I've already implemented the same approach. It works well, new and old
> tests have passed.
> On Wed, Dec 12, 2018 at 3:47 PM Denis Mekhanikov  
> wrote:
> >
> > Slava,
> >
> > Interface *Service *extends *Serializable.* So, all services are supposed
> > to be serializable by the JdkMarshaller.
> > Usage of *BinaryMarshaller* or *OptimizedMarshaller* makes sense only from
> > performance point of view.
> > But I don't think, that we should try too hard to minimize the performance
> > impact of services serialization,
> > since it doesn't happen too often.
> >
> > There are some tests like *IgniteServiceConfigVariationsFullApiTest*, that
> > check, that services, which are not
> > serializable, can be successfully deployed. I think, these tests should be
> > removed.
> > It's reasonable to require serializability, since all *Services* are marked
> > as *Serializable*, as I already mentioned.
> >
> > Slava and I discussed a possibility to choose a marshaller, depending on
> > the node state.
> > If a node is already connected to the cluster, then it could use a binary
> > marshaller,
> > otherwise JDK marshaller could be used.
> > I think, if we decide to do so, then it will complicate logic and confuse
> > users.
> > This problem exists only for static services. They are not different from
> > dynamic ones,
> > except for the way of configuration, and a moment of deployment.
> > I don't see, why different constraints should be applied to them.
> >
> > So, I'm for using the *JdkMarshaller* regardless of the service type or a
> > node state.
> >
> > Denis
> >
> > пт, 7 дек. 2018 г. в 15:57, Vyacheslav Daradur :
> >
> > > Igniters, I need your advice about the following problem:
> > >
> > > It is necessary to serialize an object (just convert an object to
> > > bytes array) for including it in joining node data (DiscoveryDataBag)
> > > *at node startup routine*.
> > >
> > > The marshalling hangs If we use 'BinaryMarshaller' or
> > > 'OptimizedMarshaler' because class can't be registered in
> > > MarshallerContextImpl#registerClassName -> transport.proposeMapping on
> > > account of the request which can't be sent through discovery-spi at
> > > the moment.
> > >
> > > Also, 'JdkMarshaller' can't be used because it imposes limits on
> > > objects that should implement 'Serializable' interface. But this
> > > restriction is unacceptable for the case.
> > >
> > > As a workaround solution, an external library, like KRYO, can be used.
> > >
> > > What tools also available in the project to solve this problem?
> > >
> > > --
> > > Best Regards, Vyacheslav D.
> > >
>
>
>
> --
> Best Regards, Vyacheslav D.



-- 
Best Regards, Vyacheslav D.