[jira] [Created] (IGNITE-9596) Web console: invisible checkbox is visible
Ilya Borisov created IGNITE-9596: Summary: Web console: invisible checkbox is visible Key: IGNITE-9596 URL: https://issues.apache.org/jira/browse/IGNITE-9596 Project: Ignite Issue Type: Bug Components: wizards Environment: Firefox, Chrome (macOS), Safari Reporter: Ilya Borisov Assignee: Alexander Kalinin Attachments: image-2018-09-14-10-58-23-054.png How it looks: !image-2018-09-14-10-58-23-054.png! How to reproduce: 1. Connect to a cluster 2. Open queries 3. Click "+Add scan" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9595) Web console: invisible checkbox is visible inqueries
Ilya Borisov created IGNITE-9595: Summary: Web console: invisible checkbox is visible inqueries Key: IGNITE-9595 URL: https://issues.apache.org/jira/browse/IGNITE-9595 Project: Ignite Issue Type: Bug Components: wizards Reporter: Ilya Borisov Assignee: Alexander Kalinin How to reproduce: 1. Connect to a cluster 2. Go to queries 3. Click on "+Add scan" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9594) Regression in release build for ignite-zookeeper module
Alexey Kuznetsov created IGNITE-9594: Summary: Regression in release build for ignite-zookeeper module Key: IGNITE-9594 URL: https://issues.apache.org/jira/browse/IGNITE-9594 Project: Ignite Issue Type: Bug Reporter: Alexey Kuznetsov Assignee: Alexey Kuznetsov Fix For: 2.7 In IGNITE-9073 from pom.xml of ignite-zookeeper jackson-core-asl & jackson-mapper-asl were removed and that leads to ClassNotFound errors at runtime if ignite node uses TcpDiscoveryZookeeperIpFinder and started from binary build. We need to return that dependencies back, because Ignite binary build logic ignore transient dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Ignite Spark Bugs
Hello, Denis. I've created a tests for Ignite codebase and attached it to the ticket [1]. [1] https://issues.apache.org/jira/browse/IGNITE-9593 В Вт, 11/09/2018 в 13:34 +0300, Nikolay Izhikov пишет: > Hello, Denis. > > Thanks for providing reproducers for bugs! > > I taked a look at your project and be able to reproduce some of errors. > I will create a ticket and start investigation in a next few days. > > В Вт, 11/09/2018 в 12:28 +0300, Dmitriy Pavlov пишет: > > Hi Denis, > > > > Thank you for bringing this here and for your efforts to reproduce it. > > Would you like to create an issue and contribute these test into Ignite > > code base? > > > > If you would like to proceed with the patch submission process, please > > sign-in to Apache JIRA and share your JIRA username, I will add you as the > > contributor. > > > > You can count on my support. > > > > Sincerely, > > Dmitriy Pavlov > > > > вт, 11 сент. 2018 г. в 10:51, Денис Костин : > > > > > Hello everybody! > > > > > > I found a few bugs in the actual version of Apache Ignite Spark (2.6.0) > > > and described in my GitHub: https://github.com/x-x-z/IgniteSparkBugs > > > > > > I wrote a test for each case: > > > https://github.com/x-x-z/IgniteSparkBugs/blob/master/src/test/java/org/xxz/ignite_spark_bugs/IgniteSparkBugsApplicationTests.java > > > > > > Check, please. > > > > > > > > > Best regards, > > > Denis Kostin signature.asc Description: This is a digitally signed message part
[jira] [Created] (IGNITE-9593) Spart Optimization fails to optimize statements
Nikolay Izhikov created IGNITE-9593:
---
Summary: Spart Optimization fails to optimize statements
Key: IGNITE-9593
URL: https://issues.apache.org/jira/browse/IGNITE-9593
Project: Ignite
Issue Type: Bug
Components: spark
Affects Versions: 2.6
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov
Fix For: 2.7
Attachments: Spark_optimization_bugs_reproducer.patch
In some cases, {{IgniteOptimization}} fails to optimize spark query. Reproducer
attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Python thin client
Igor, The commented code (lines 95-96) gives an error if executed. The error is stated just below, in lines 98-100. It is explained here: https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html#create I found out by trial and error that a cache, created with SQL DDL statement (`CREATE TABLE`), can be deleted only with SQL DDL statement (`DROP TABLE`), whereas a table, created with a binary protocol operation (`OP_CACHE_CREATE_*` or `OP_CACHE_GET_OR_CREATE_*`) can be deleted only with binary protocol operation (`OP_CACHE_DESTROY`). I miss this fact in Ignite documentation. In this particular part of the example I try to stress this out with regard to the cache created in the beginning of the example. The cache behaves like an SQL table, but can not be dropped. You need to destroy it instead (line 102). I would not go into such depths at all, but the examples are designed to be runnable in automated environments, and thus must have some cleanup code. And since the cleanup here is not absolutely trivial, I thought I must explain it. I also could handle an error instead of commenting the erroneous code, like ``` any_error = None try: DROP_QUERY = 'DROP TABLE Student' client.sql(DROP_QUERY) except Exception as e: any_error = e print(any_error) # pyignite.exceptions.SQLError: class org.apache.ignite.IgniteCheckedException: # Only cache created with CREATE TABLE may be removed with DROP TABLE # [cacheName=SQL_PUBLIC_STUDENT] but it would complicate the cleanup part of the example to the point it is lost any exemplariness. So I decided to simply use comments. On 9/14/18 2:48 AM, Igor Sapego wrote: Ok, now everything's running. API looks good to me. I have a single question about example code: What these comments are for - [1]? [1] - https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/create_binary.py#L95 Best Regards, Igor On Thu, Sep 13, 2018 at 12:52 AM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: Igor, Yes, it's my bad, sorry. Just merged the Ignite master with my branch. On 9/12/18 8:47 PM, Igor Sapego wrote: Pavel, Yes, I did. I tried completely clean environment, followed the same steps and got the same error. Then I removed attr, and out of sudden everything started working. Dmitry, Thanks, now it's more clear: Handshake error: Unsupported version. Server expects binary protocol version 1.1.0. Client provides 1.2.0. Why do you use version 1.2.0, but have outdated server code? I propose you to merge with or rebase onto Ignite's master branch. Best Regards, Igor On Tue, Sep 11, 2018 at 11:21 PM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: Igor, I have just commited an improvment to the HandshakeError message generation algorithm. I hope it is now easier to understand what expects what in case of binary protocol version mismatch. Thank you for pointing this out. On 9/12/18 2:13 AM, Igor Sapego wrote: I managed to start tests, and now I'm getting the following message: pyignite.exceptions.HandshakeError: Handshake error: Unsupported version. Expected protocol version: 1.1.0. It would be useful to print "Unexpected version" itself, because I can not understand what is the issue. Best Regards, Igor
[jira] [Created] (IGNITE-9592) MVCC: Use linked lists to store multiple versions.
Roman Kondakov created IGNITE-9592:
--
Summary: MVCC: Use linked lists to store multiple versions.
Key: IGNITE-9592
URL: https://issues.apache.org/jira/browse/IGNITE-9592
Project: Ignite
Issue Type: Improvement
Components: mvcc
Reporter: Roman Kondakov
Currently we store all versions of each row in primary index. It is not very
efficient for several reasons:
* We have to insert and remove tree item each time a new version is created or
an old version deleted. This leads to a considerable tree operations number
increasing as well as tree splits and merges operations.
* Also this leads to a contention on leaf page write lock - each update
operation has to obtain exclusive access to insert a new version of row. During
this update no body on that leaf can not be able to update or even read data of
neighbour keys.
* Primary key tree consumes more space if it stores each version.
* Other vendors do not store each version in primary indexes (as far as I
know).
Possible solution - store only key and link to the newest version in the
primary index. Instead of this {{CacheDataTree}} item
{{| key part | | |}}
{{|-| lockVer | link |}}
{{| cache_id | hash | mvccVer | | |}}
we'll have:
{{| key part | | link to the |}}
{{|-| lockVer | newest |}}
{{| cache_id | hash | |version |}}
Note, we do not have mvccVer in tree item. Link to the newer version leads to
the most recent inserted data item. To find older versions, each DatRow is
provided with "prevLink" - the link to previous version of row. DataRow layout
can be changed from
| header | xid_max | xid_min | KV bytes |
to the next one:
| header | xid_max | xid_min | *PREV_LINK* | KV bytes |
Where *PREV_LINK* field points to the previous version of the row. Traversing
this prevRow links we can iterate over all available versions without affecting
primary key tree.
When the new version is created we just insert the new row in datastore, then
do CAS on the link to the newest version in primary key tree in order it points
to the new row. PrevLink in the new row should point on the previous one. That
is all. We've just inserted a new version just with a long field CAS in the
CacheDataTree. Without obtaining write lock and other headache.
Secondary indexes are handled in the same manner as before.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-9591) MVCC: Do small write operations under the read lock.
Roman Kondakov created IGNITE-9591:
--
Summary: MVCC: Do small write operations under the read lock.
Key: IGNITE-9591
URL: https://issues.apache.org/jira/browse/IGNITE-9591
Project: Ignite
Issue Type: Improvement
Components: mvcc
Reporter: Roman Kondakov
There are several operations in MVCC flow which can be done under the page read
lock instead of write lock. For example, setting xid_max on a datarow. It is
safe, because races are possible only between concurrent transactions, which
are not visible to each other.
See {{MvccMarkUpdatedHandler}} and other similar handlers in this class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-9590) MVCC: Cleanup old rows in the vacuum thread.
Roman Kondakov created IGNITE-9590: -- Summary: MVCC: Cleanup old rows in the vacuum thread. Key: IGNITE-9590 URL: https://issues.apache.org/jira/browse/IGNITE-9590 Project: Ignite Issue Type: Improvement Components: mvcc Reporter: Roman Kondakov When updating writer thread should iterate over the set of the last versions in order to find an appropriate version for its MVCC snapshot. During this iteration it collects invisible to anybody versions (their xid_max version is less than cleanup). When all outdated versions found, writer thread cleanups these rows - removes it from indexes and from pagestore. It would be more efficient if writer thread does not cleanup old rows by itself, but rather delegate it to vacuum workers: instead of cleaning just put it to cleanup queue. in case of significant backpressure during active updates, when cleanup workers can't keep up with removing outdated rows and cleanup queue is too big, writer threads can cleanup this rows by itself to prevent memory and performance problems. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: TeamCity Helper shows new failures in PRs
Hello, Dmitrii. Can you give a link to detailed description of this feature? Actually, I wonder if someone except bit developers know how to use it чт, 13 сент. 2018 г., 22:46 Dmitrii Ryabov : > Hello, Igniters! > > We've improved tests analysis in the TeamCity Helper. > > Now, when you view PR analysis, you can see a table with possible blockers > - failed tests and suites, that most likely appeared in the pull request. >
Re: The future of Affinity / Topology concepts and possible PME optimizations.
Ilya, Thanks for the detailed explanation. Everything you suggested makes sense. Needless to say, PME effect on the grid should be minimized. Let's start with the simpler and less risky changes first. D. On Thu, Sep 13, 2018 at 5:58 AM Ilya Lantukh wrote: > Igniters, > > As most of you know, Ignite has a concept of AffinityTopologyVersion, which > is associated with nodes that are currently present in topology and a > global cluster state (active/inactive, baseline topology, started caches). > Modification of either of them involves process called Partition Map > Exchange (PME) and results in new AffinityTopologyVersion. At that moment > all new cache and compute grid operations are globally "frozen". This might > lead to indeterminate cache downtimes. > > However, our recent changes (esp. introduction of Baseline Topology) caused > me to re-think those concept. Currently there are many cases when we > trigger PME, but it isn't necessary. For example, adding/removing client > node or server node not in BLT should never cause partition map > modifications. Those events modify the *topology*, but *affinity* in > unaffected. On the other hand, there are events that affect only *affinity* > - most straightforward example is CacheAffinityChange event, which is > triggered after rebalance is finished to assign new primary/backup nodes. > So the term *AffinityTopologyVersion* now looks weird - it tries to "merge" > two entities that aren't always related. To me it makes sense to introduce > separate *AffinityVersion *and *TopologyVersion*, review all events that > currently modify AffinityTopologyVersion and split them into 3 categories: > those that modify only AffinityVersion, only TopologyVersion and both. It > will allow us to process such events using different mechanics and avoid > redundant steps, and also reconsider mapping of operations - some will be > mapped to topology, others - to affinity. > > Here is my view about how different event types theoretically can be > optimized: > 1. Client node start / stop: as stated above, no PME is needed, ticket > https://issues.apache.org/jira/browse/IGNITE-9558 is already in progress. > 2. Server node start / stop not from baseline: should be similar to the > previous case, since nodes outside of baseline cannot be partition owners. > 3. Start node in baseline: both affinity and topology versions should be > incremented, but it might be possible to optimize PME for such case and > avoid cluster-wide freeze. Partition assignments for such node are already > calculated, so we can simply put them all into MOVING state. However, it > might take significant effort to avoid race conditions and redesign our > architecture. > 4. Cache start / stop: starting or stopping one cache doesn't modify > partition maps for other caches. It should be possible to change this > procedure to skip PME and perform all necessary actions (compute affinity, > start/stop cache contexts on each node) in background, but it looks like a > very complex modification too. > 5. Rebalance finish: it seems possible to design a "lightweight" PME for > this case as well. If there were no node failures (and if there were, PME > should be triggered and rebalance should be cancelled anyways) all > partition states are already known by coordinator. Furthermore, no new > MOVING or OWNING node for any partition is introduced, so all previous > mappings should still be valid. > > For the latter complex cases in might be necessary to introduce "is > compatible" relationship between affinity versions. Operation needs to be > remapped only if new version isn't compatible with the previous one. > > Please share your thoughts. > > -- > Best regards, > Ilya >
TeamCity Helper shows new failures in PRs
Hello, Igniters! We've improved tests analysis in the TeamCity Helper. Now, when you view PR analysis, you can see a table with possible blockers - failed tests and suites, that most likely appeared in the pull request.
[GitHub] ignite pull request #4754: Ignite 9349
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4754 ---
[GitHub] ignite pull request #4700: IGNITE-9212 Uncomment or explain various commente...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4700 ---
[GitHub] ignite pull request #4754: Ignite 9349
GitHub user ignitetcbot opened a pull request: https://github.com/apache/ignite/pull/4754 Ignite 9349 You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/ignite ignite-9349 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4754.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4754 commit 3fee44b0189e50a215e501cd7c7c43b08bf404a3 Author: Dmitriy Pavlov Date: 2018-08-22T15:05:39Z IGNITE-9349 Spring Data 2x and 1x and Spring 5x and 4x versions were updated commit ef963cc53d5575e5051accdfafe161d7f6636294 Author: Dmitriy Pavlov Date: 2018-09-12T18:28:53Z IGNITE-9349 Spring Data 2x and 1x and Spring 5x and 4x versions were updated commit 13cb990af44043ccb0e63fb848a84598e4c6ac90 Author: Dmitriy Pavlov Date: 2018-09-13T12:47:23Z Merge branch 'master' into ignite-9349 commit 1b6cda4a142f89156b64a8d1a0f0dbdb91d23655 Author: Dmitriy Pavlov Date: 2018-09-13T16:10:21Z IGNITE-9349 Spring Data 2x and 1x and Spring 5x and 4x versions were updated ---
[GitHub] ignite pull request #4753: IGNITE-9585 Error message sometimes refers nonexi...
GitHub user oignatenko opened a pull request: https://github.com/apache/ignite/pull/4753 IGNITE-9585 Error message sometimes refers nonexisting log file when remote node fails to start You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9585 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4753.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4753 commit 1eeca908a8076a8317947dac8a46845964d1d7ea Author: Oleg Ignatenko Date: 2018-08-23T13:13:28Z IGNITE-9348 ML examples improvements - wip (logging improved) -- verified with diffs overview, executing the examples and studying execution logs commit e50b89c392568ba9b93935c4fa6c7f7f93f5ec6f Author: Oleg Ignatenko Date: 2018-08-23T14:45:57Z Revert "IGNITE-9348 ML examples improvements" This reverts commit 1eeca908a8076a8317947dac8a46845964d1d7ea. commit 474024b4c5bbdb3c0a4ed2f0a66238c8054c6674 Author: Oleg Ignatenko Date: 2018-08-23T14:57:34Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 9642b233b5701bdad47ebea163079160227c582a Author: Oleg Ignatenko Date: 2018-08-28T14:01:11Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 7fc16d013ab725d2ff2e1a1b042c983f11d0c4d4 Author: Oleg Ignatenko Date: 2018-08-28T15:13:02Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit d2caba67b156674f051f50faebeafe0871bf0914 Author: Oleg Ignatenko Date: 2018-08-29T13:14:07Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 16775dff51d71ea68b4a3dea98be552130c493ed Author: Oleg Ignatenko Date: 2018-08-30T09:00:56Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit aedb59929974fe205b949225c1a338c68c60cfc8 Author: Oleg Ignatenko Date: 2018-08-30T09:42:38Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 39c6482fcdca506aa33011ed21c98060b4a8c68b Author: Oleg Ignatenko Date: 2018-08-30T11:28:05Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 33b32a2760a6559c78283b927e3191180d8ed9e1 Author: Oleg Ignatenko Date: 2018-08-30T12:31:16Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 9531028ddd1aef9e95f7e8c8b528086739bbb1b0 Author: Oleg Ignatenko Date: 2018-08-30T14:06:34Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 28f22c6e2fffcb82717ba5da7be2cfd39715c4e3 Author: Oleg Ignatenko Date: 2018-08-30T16:41:51Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit aacac88db519187413b0fc5ff9d0e55b8f8cff22 Author: Oleg Ignatenko Date: 2018-08-31T10:12:32Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 897f920dde46022849b13f9fb86dba8e54119a56 Author: Oleg Ignatenko Date: 2018-08-31T13:57:14Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 114c79e54c1b316006ccc3ff22d20d902f9313df Author: Oleg Ignatenko Date: 2018-08-31T17:39:16Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit fd6b659bb8be1992618ce4ce91f568a0988b3afa Author: Oleg Ignatenko Date: 2018-09-02T06:11:42Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 6ae6d1d3cf9743d8d466be0330511ddc8589e944 Author: Oleg Ignatenko Date: 2018-09-03T10:27:35Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit e8b27dbd3d0c1cbdb7a7659175f5c7bb447482bf Author: Oleg Ignatenko Date: 2018-09-04T09:49:44Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 622c82efdd0aa182fadea6b7ffa5d4615521a3f5 Author: Oleg Ignatenko Date: 2018-09-05T10:50:28Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit fb844fe3751e2e8ae87e6b8030b0e4bd659df9c2 Author: Oleg Ignatenko Date: 2018-09-05T11:45:58Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 480ed78869277d7e32f725ab71bec9621f1ac03a Author: Oleg Ignatenko Date: 2018-09-06T07:52:55Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit c99762498f617c0e98ea3062a43c0b30092166ef Author: Oleg Ignatenko Date: 2018-09-06T14:45:04Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 2e17175225c72f747d370b5fee96f8be69d6d2cb Author: Oleg Ignatenko Date: 2018-09-06T17:33:54Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 9ebcd9a2fe5966b0bf42a95484395867c15d863f Author: Oleg Ignatenko Date: 2018-09-07T13:38:51Z Merge
[GitHub] zzzadruga opened a new pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval
zzzadruga opened a new pull request #9: IGNITE-9541 Add the comparison for two general statistics "RunAll" for master in the date interval URL: https://github.com/apache/ignite-teamcity-bot/pull/9 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ignite pull request #4752: IGNITE-8006 added parallel start of caches.
GitHub user akalash opened a pull request: https://github.com/apache/ignite/pull/4752 IGNITE-8006 added parallel start of caches. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-8006 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4752 commit a457c8b4c23be57cea11f7f7a49c5bc695adde9d Author: Anton Kalashnikov Date: 2018-09-13T16:44:40Z IGNITE-8006 added parallel start of caches. ---
Re: Python thin client
Ok, now everything's running. API looks good to me. I have a single question about example code: What these comments are for - [1]? [1] - https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/create_binary.py#L95 Best Regards, Igor On Thu, Sep 13, 2018 at 12:52 AM Dmitry Melnichuk < dmitry.melnic...@nobitlost.com> wrote: > Igor, > > Yes, it's my bad, sorry. Just merged the Ignite master with my branch. > > On 9/12/18 8:47 PM, Igor Sapego wrote: > > Pavel, > > Yes, I did. I tried completely clean environment, followed the same > > steps and got the same error. Then I removed attr, and out of sudden > > everything started working. > > > > Dmitry, > > Thanks, now it's more clear: > > Handshake error: Unsupported version. Server expects binary protocol > > version 1.1.0. Client provides 1.2.0. > > > > Why do you use version 1.2.0, but have outdated server code? I > > propose you to merge with or rebase onto Ignite's master branch. > > > > Best Regards, > > Igor > > > > > > On Tue, Sep 11, 2018 at 11:21 PM Dmitry Melnichuk < > > dmitry.melnic...@nobitlost.com> wrote: > > > >> Igor, > >> > >> I have just commited an improvment to the HandshakeError message > >> generation algorithm. I hope it is now easier to understand what expects > >> what in case of binary protocol version mismatch. > >> > >> Thank you for pointing this out. > >> > >> On 9/12/18 2:13 AM, Igor Sapego wrote: > >>> I managed to start tests, and now I'm getting the following message: > >>> > >>> pyignite.exceptions.HandshakeError: Handshake error: Unsupported > >>> version. Expected protocol version: 1.1.0. > >>> > >>> It would be useful to print "Unexpected version" itself, because I can > >>> not understand what is the issue. > >>> > >>> Best Regards, > >>> Igor > >> > > > >
[GitHub] ignite pull request #4751: IGNITE-9589
GitHub user NSAmelchev opened a pull request: https://github.com/apache/ignite/pull/4751 IGNITE-9589 For IGNITE-9589 GridTcpCommunicationSpiConfigSelfTest.testLocalPortRange You can merge this pull request into a Git repository by running: $ git pull https://github.com/NSAmelchev/ignite ignite-9589 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4751.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4751 commit f6f75ae26f744cd9ac2a376c1f5735b1016f Author: NSAmelchev Date: 2018-09-13T16:40:12Z Check that test fails on mass runs ---
[jira] [Created] (IGNITE-9589) GridTcpCommunicationSpiConfigSelfTest.testLocalPortRange is flaky in master
Amelchev Nikita created IGNITE-9589:
---
Summary: GridTcpCommunicationSpiConfigSelfTest.testLocalPortRange
is flaky in master
Key: IGNITE-9589
URL: https://issues.apache.org/jira/browse/IGNITE-9589
Project: Ignite
Issue Type: Bug
Reporter: Amelchev Nikita
Assignee: Amelchev Nikita
The test GridTcpCommunicationSpiConfigSelfTest.testLocalPortRange fails
periodicaly.
>From the logs, I see that the failure is caused by BindException. It causes
>node start fails because the test port range is 0.
{noformat}
[2018-09-13
04:06:20,060][ERROR][test-runner-#225862%tcp.GridTcpCommunicationSpiConfigSelfTest%][GridTcpCommunicationSpiConfigSelfTest]
Failed to start manager: GridManagerAdapter [enabled=true,
name=o.a.i.i.managers.communication.GridIoManager]
class org.apache.ignite.IgniteCheckedException: Failed to get SPI attributes.
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:278)
at
org.apache.ignite.internal.managers.communication.GridIoManager.start(GridIoManager.java:262)
at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1755)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:975)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2020)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1725)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1153)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:673)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:598)
at org.apache.ignite.Ignition.start(Ignition.java:323)
at
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:1370)
at
org.apache.ignite.spi.communication.tcp.GridTcpCommunicationSpiConfigSelfTest.testLocalPortRange(GridTcpCommunicationSpiConfigSelfTest.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2177)
at
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:143)
at
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2092)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to initialize
TCP server: 0.0.0.0/0.0.0.0
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:2137)
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:261)
... 20 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to bind to
any port within range [startPort=47100, portRange=0, locHost=0.0.0.0/0.0.0.0]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:2450)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.getNodeAttributes(TcpCommunicationSpi.java:2134)
... 21 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to initialize
NIO selector.
at
org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:988)
at
org.apache.ignite.internal.util.nio.GridNioServer.(GridNioServer.java:342)
at
org.apache.ignite.internal.util.nio.GridNioServer.(GridNioServer.java:97)
at
org.apache.ignite.internal.util.nio.GridNioServer$Builder.build(GridNioServer.java:3669)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.resetNioServer(TcpCommunicationSpi.java:2415)
... 22 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at
org.apache.ignite.internal.util.nio.GridNioServer.createSelector(GridNioServer.java:972)
... 26 more
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-9588) Separate page for JIRA, GitHub actions
Ryabov Dmitrii created IGNITE-9588: -- Summary: Separate page for JIRA, GitHub actions Key: IGNITE-9588 URL: https://issues.apache.org/jira/browse/IGNITE-9588 Project: Ignite Issue Type: Sub-task Reporter: Ryabov Dmitrii Assignee: Ryabov Dmitrii To separate JIRA and GitHub actions from other action on index page we need to create an additional page, opened by tab on the panel with Home and Compare Builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Workflow improvement
Yep, it's working. Thank you. 2018-09-13 19:09 GMT+03:00 Dmitriy Pavlov : > Hi Dmitriy, > > Sure, I agree with extra permissions assignment. Done. > > Could you please check if you can manage your builds now? > > Sincerely, > Dmitriy Pavlov > > чт, 13 сент. 2018 г. в 16:05, Dmitrii Ryabov : > > > Hi, Dmitriy, > > > > Can you give me rights to stop my builds on TeamCity? Working on the > TCH, I > > run a lot of builds, and I don't like to ask other people to stop builds > > too often. > > > > 2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov : > > > > > Hi, Dmitriy, > > > > > > I made PRs in my fork for test purposes. Real PRs were made to the > GitHub > > > mirror, and one of them is already merged by D. Govorukhin. PR with > > GitHub > > > statuses [1] is ready for review. PR with JIRA comment will be ready > in a > > > few days. > > > > > > [1] https://github.com/apache/ignite-teamcity-bot/pull/5 > > > > > > 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov : > > > > > >> Hi Dmitrii, > > >> > > >> I deployed change with blockers summary of failures at the top of PR > > >> result > > >> page. The Bot is migrating entries now, I hope it will be done in 1-4 > > >> hours. > > >> > > >> I noticed your PRs are created in your own fork, not in > > >> https://github.com/apache/ignite-teamcity-bot > > >> Could you please create unmerged PR in GitHub mirror so it could be > > merged > > >> after reviewing by the apply-pr script. > > >> > > >> Sincerely, > > >> Dmitriy Pavlov > > >> > > >> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov : > > >> > > >> > Hi, Dmitriy, > > >> > > > >> > I created a table with possible blockers [1] for the page with the > > >> latest > > >> > build analysis. Anton K. has reviewed it. Can you check and merge > it? > > >> > > > >> > Also, I created a button to notify PR about analisis results [2]. I > > used > > >> > GitHub statuses (example [3]), but to work it needs a token with > push > > >> > permission. As Apache PMC, can you acquire such token? I think > > statuses > > >> are > > >> > much more notable than comments. > > >> > > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-9376 > > >> > [2] https://issues.apache.org/jira/browse/IGNITE-9377 > > >> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls > > >> > > > >> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov : > > >> > > > >> > > Not sure I understood what bot statistic reduction means. > > >> > > > > >> > > As far as I understood, you also mean locating failure > automatically > > >> with > > >> > > re-runs on specific git revisions (hashes). I found it will be a > > good > > >> > > contribution to TC Bot for both flaky and non-flaky failures > > >> detection. > > >> > E.g > > >> > > failure occurred after 10 commits merged to master. How to find > out > > >> bad > > >> > > commit? Automatic location (analog of bisecting, but using TC > test) > > >> will > > >> > be > > >> > > really helpful. But it is not a simple contribution. > > >> > > > > >> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov < > somefire...@gmail.com > > >: > > >> > > > > >> > > > Hi, Dmitriy, > > >> > > > > > >> > > > Good, I'll create tickets for this steps. > > >> > > > > > >> > > > Stable suites are a good idea, but also we need to do some > actions > > >> > when a > > >> > > > flaky test will appear in stable suite of master branch run. To > > find > > >> > out > > >> > > a > > >> > > > guilty commit, we should run previous commits on empty > TeamCity's > > >> > queue. > > >> > > > This runs will reduce bot's statistics for the latest commits. > > >> > > > > > >> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov < > dpavlov@gmail.com > > >: > > >> > > > > > >> > > > > Hi Dmitriy, > > >> > > > > > > >> > > > > Sounds like a plan ;) I totally agree. > > >> > > > > Just one proposal: I would like to avoid hiding any test > > >> failures. 2 > > >> > > > > separate tables named 'Possible Blockers' and 'Other failures' > > >> should > > >> > > be > > >> > > > > completely clear. Comment to PR can contain only first > category. > > >> > > > > > > >> > > > > We had a private discussion with Anton K. and he proposed a > very > > >> > > > > interesting thing, I would like to bring it here. We can add > > >> > > > configuration > > >> > > > > into TC bot and mark some tests and some suites as 'Known > > Stable' > > >> It > > >> > > > means > > >> > > > > that suite or test failure should be considered as a blocker > for > > >> > merge > > >> > > > > every time it fails, even if fail rate is non-zero. Very first > > >> list > > >> > of > > >> > > > such > > >> > > > > suites are > > >> > > > > - Build Apache Ignite > > >> > > > > - License check > > >> > > > > And tests: > > >> > > > > - .Net API Parity check > > >> > > > > Please share your vision. > > >> > > > > > > >> > > > > Meanwhile, I've updated the TC bot. > > >> > > > > - Underlying Apache Ignite DB was refactored to use lower > > >> partitions > > >> > > > count, > > >> > > > > so restart should be
[GitHub] ignite pull request #4750: IGNITE-8770: OutOfMemory in Queries1 suite in mas...
GitHub user avplatonov opened a pull request: https://github.com/apache/ignite/pull/4750 IGNITE-8770: OutOfMemory in Queries1 suite in master branch on TC You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-8770 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4750.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4750 commit 72994e61f554b67c01ad3439af95e263e8b75cab Author: Alexey Platonov Date: 2018-09-13T16:08:12Z limit partitions count ---
Re: Workflow improvement
Hi Dmitriy, Sure, I agree with extra permissions assignment. Done. Could you please check if you can manage your builds now? Sincerely, Dmitriy Pavlov чт, 13 сент. 2018 г. в 16:05, Dmitrii Ryabov : > Hi, Dmitriy, > > Can you give me rights to stop my builds on TeamCity? Working on the TCH, I > run a lot of builds, and I don't like to ask other people to stop builds > too often. > > 2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov : > > > Hi, Dmitriy, > > > > I made PRs in my fork for test purposes. Real PRs were made to the GitHub > > mirror, and one of them is already merged by D. Govorukhin. PR with > GitHub > > statuses [1] is ready for review. PR with JIRA comment will be ready in a > > few days. > > > > [1] https://github.com/apache/ignite-teamcity-bot/pull/5 > > > > 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov : > > > >> Hi Dmitrii, > >> > >> I deployed change with blockers summary of failures at the top of PR > >> result > >> page. The Bot is migrating entries now, I hope it will be done in 1-4 > >> hours. > >> > >> I noticed your PRs are created in your own fork, not in > >> https://github.com/apache/ignite-teamcity-bot > >> Could you please create unmerged PR in GitHub mirror so it could be > merged > >> after reviewing by the apply-pr script. > >> > >> Sincerely, > >> Dmitriy Pavlov > >> > >> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov : > >> > >> > Hi, Dmitriy, > >> > > >> > I created a table with possible blockers [1] for the page with the > >> latest > >> > build analysis. Anton K. has reviewed it. Can you check and merge it? > >> > > >> > Also, I created a button to notify PR about analisis results [2]. I > used > >> > GitHub statuses (example [3]), but to work it needs a token with push > >> > permission. As Apache PMC, can you acquire such token? I think > statuses > >> are > >> > much more notable than comments. > >> > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-9376 > >> > [2] https://issues.apache.org/jira/browse/IGNITE-9377 > >> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls > >> > > >> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov : > >> > > >> > > Not sure I understood what bot statistic reduction means. > >> > > > >> > > As far as I understood, you also mean locating failure automatically > >> with > >> > > re-runs on specific git revisions (hashes). I found it will be a > good > >> > > contribution to TC Bot for both flaky and non-flaky failures > >> detection. > >> > E.g > >> > > failure occurred after 10 commits merged to master. How to find out > >> bad > >> > > commit? Automatic location (analog of bisecting, but using TC test) > >> will > >> > be > >> > > really helpful. But it is not a simple contribution. > >> > > > >> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov >: > >> > > > >> > > > Hi, Dmitriy, > >> > > > > >> > > > Good, I'll create tickets for this steps. > >> > > > > >> > > > Stable suites are a good idea, but also we need to do some actions > >> > when a > >> > > > flaky test will appear in stable suite of master branch run. To > find > >> > out > >> > > a > >> > > > guilty commit, we should run previous commits on empty TeamCity's > >> > queue. > >> > > > This runs will reduce bot's statistics for the latest commits. > >> > > > > >> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov >: > >> > > > > >> > > > > Hi Dmitriy, > >> > > > > > >> > > > > Sounds like a plan ;) I totally agree. > >> > > > > Just one proposal: I would like to avoid hiding any test > >> failures. 2 > >> > > > > separate tables named 'Possible Blockers' and 'Other failures' > >> should > >> > > be > >> > > > > completely clear. Comment to PR can contain only first category. > >> > > > > > >> > > > > We had a private discussion with Anton K. and he proposed a very > >> > > > > interesting thing, I would like to bring it here. We can add > >> > > > configuration > >> > > > > into TC bot and mark some tests and some suites as 'Known > Stable' > >> It > >> > > > means > >> > > > > that suite or test failure should be considered as a blocker for > >> > merge > >> > > > > every time it fails, even if fail rate is non-zero. Very first > >> list > >> > of > >> > > > such > >> > > > > suites are > >> > > > > - Build Apache Ignite > >> > > > > - License check > >> > > > > And tests: > >> > > > > - .Net API Parity check > >> > > > > Please share your vision. > >> > > > > > >> > > > > Meanwhile, I've updated the TC bot. > >> > > > > - Underlying Apache Ignite DB was refactored to use lower > >> partitions > >> > > > count, > >> > > > > so restart should be faster. > >> > > > > - Now 100 builds are saved as our failure rate statistics basis. > >> > > > Flakiness > >> > > > > border was not changed, so more test now will be considered as > >> flaky. > >> > > > > - Now 4 builds required to be failed or timed out in a row > before > >> > > > > notification is sent. > >> > > > > > >> > > > > Sincerely, > >> > > > > Dmitriy Pavlov > >> > > > > > >> > > > > чт, 23 авг. 2018 г. в 17:09,
[ML] [New Feature] Handle different labels in training data and handle unknown labels in test or updated training data correctly
Hi, Igniters Welcome to the discussion about labels handling during ML training. The problem is that all algorithms of binary classification are ready to handle the datasets marked with 0/1 labels and predict 0/1 labels without especial mapping. Also the algorithms don't handle situation with unknown labels during the updating and testing phases Umbrella ticket is created here https://issues.apache.org/jira/browse/IGNITE-9587 Also, I'd invite you to discuss 1) list of trainers to upgrade with this feature 2) how to handle unknown labels during the prediction/test phase 3) how to handle unknown labels during the update model phase (new data is coming during the next training and the results of next training should be merged with the results of the previous training) 4) Where to store metadata about labels during training phases Glad to hear your ideas
[jira] [Created] (IGNITE-9587) [ML] Umbrella ticket: Handle different labels in training data and handle unknown labels in test or updated training data correctly
Aleksey Zinoviev created IGNITE-9587: Summary: [ML] Umbrella ticket: Handle different labels in training data and handle unknown labels in test or updated training data correctly Key: IGNITE-9587 URL: https://issues.apache.org/jira/browse/IGNITE-9587 Project: Ignite Issue Type: New Feature Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev The problem is that all algorithms of binary classification are ready to handle the datasets marked with 0/1 labels and predict 0/1 labels without especial mapping. Also the algorithms don't handle situation with unknown labels during the updating and testing phases Possible solution: it could be stored in context of ML training -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: IoC/DI support in Apache Ignite.NEt
Looks good to me in general.
But we should add IDependencyResolver {get;set} to IgniteConfiguration
class instead of changing IIgnite interface.
We need the resolver at startup before IIgnite instance is created, to
resolve user-defined cache stores and so on.
And we should make sure to add tests for all kinds of user-defined classes
that are instantiated by Ignite:
Compute jobs, actions, services, cache stores, ICacheEntryFilter,
ICacheEntryProcessor, etc.
Thanks,
Pavel
On Thu, Sep 13, 2018 at 4:03 PM Artyom Sokolov wrote:
> Hello,
>
> I propose the following.
>
> *Development*
>
> 1. Define *IDependencyResolver* in *Apache.Ignite.Core* under
> *DependencyInjection* folder that contains at least 2 methods:
>
> *T Resolve(object arguments = null);*
> *object Resolve(Type type, object arguments = null);*
>
> 2. Create *InjectResolvableAttribute* in Apache.Ignite.Core under
> *DependencyInjection* folder marked with
> *[AttributeUsage(AttributeTargets.Field
> | AttributeTargets.Property)]*
> 3. Create internal static class *DependencyInjector* with at least one
> internal static method that will inject required dependencies in service
> implementations like this one (exception checking is omitted currently and
> has to be added)
>
> *namespace Apache.Ignite.Core.DependencyInjection*
> *{*
> *using System.Linq;*
> *using System.Reflection;*
> *using Apache.Ignite.Core.Services;*
>
> *internal static class DependencyInjector*
> *{*
> *internal static void Inject(IService svc, IIgnite ignite)*
> *{*
> *var dependencyResolver = ignite.GetDependencyResolver();*
>
> *if (dependencyResolver == null)*
> *{*
> *return;*
> *}*
>
> *foreach (var property in svc.GetType()*
> *.GetProperties(BindingFlags.Public |
> BindingFlags.NonPublic | BindingFlags.Instance))*
> *{*
> *if
> (!property.GetCustomAttributes(typeof(InjectResolvableAttribute),
> true).Any())*
> *{*
> *continue;*
> *}*
>
> *if (!property.CanWrite)*
> *{*
> *continue;*
> *}*
>
> *var resolvedDependency =
> dependencyResolver.Resolve(property.PropertyType);*
> *property.SetValue(svc, resolvedDependency, null);*
> *}*
>
> *foreach (var field in svc.GetType()*
> *.GetFields(BindingFlags.Public | BindingFlags.NonPublic |
> BindingFlags.Instance))*
> *{*
> *if
> (!field.GetCustomAttributes(typeof(InjectResolvableAttribute),
> true).Any())*
> *{*
> *continue;*
> *}*
>
> *var resolvedDependency =
> dependencyResolver.Resolve(field.FieldType);*
>
> *field.SetValue(svc, resolvedDependency);*
> *}*
> *}*
> *}*
> *}*
>
> 4. Add* DependencyInjector.Inject(svc, _ignite);* call to
> *UnmanagedCallbacks.cs* right after *ResourceProcessor.Inject(svc,
> _ignite)*
> .
> 5. Add 2 methods to *Ignite* class and it's interface *IIgnite*:
>
> *public IDependencyResolver GetDependencyResolver();*
> *public IIgnite SetDependencyResolver(IDependencyResolver
> dependencyResolver);*
>
> 6. Create *Apache.Ignite.DependencyInjection.CastleWindsor* project and
> implement *IDependencyResolver* with *CastleWindsorDependencyResolver*
> class.
>
> *namespace Apache.Ignite.DependencyInjection.CastleWindsor*
> *{*
> *using System;*
> *using Castle.MicroKernel;*
> *using IDependencyResolver =
> Core.DependencyInjection.IDependencyResolver;*
>
> *public class CastleWindsorDependencyResolver : IDependencyResolver*
> *{*
> *private IKernel Kernel { get; set; }*
>
> *public CastleWindsorDependencyResolver(IKernel kernel)*
> *{*
> *Kernel = kernel;*
> *}*
>
> *public T Resolve(object arguments = null)*
> *{*
> *return arguments == null ? Kernel.Resolve() :
> Kernel.Resolve(arguments);*
> *}*
>
> *public object Resolve(Type type, object arguments = null)*
> *{*
> *return arguments == null ? Kernel.Resolve(type) :
> Kernel.Resolve(type, arguments);*
> *}*
> *}*
> *}*
>
> 7. Create extension like this one
>
> namespace Apache.Ignite.DependencyInjection.CastleWindsor
> {
> using Apache.Ignite.Core;
> using Castle.MicroKernel;
>
> public static class IgniteExtensions
> {
> public static IIgnite UseCastleWindsorResolver(this IIgnite ignite,
> IKernel kernel)
> {
> return ignite.SetDependencyResolver(new
> CastleWindsorDependencyResolver(kernel));
> }
> }
> }
>
> 8. Write test.
>
>
> *Usage*
>
> var grid = Ignition.Start();
>
> using (var container = new WindsorContainer())
> {
>
>
[jira] [Created] (IGNITE-9586) Treat 172.17.0.1 as localhost address and only use it as last resort
Ilya Kasnacheev created IGNITE-9586: --- Summary: Treat 172.17.0.1 as localhost address and only use it as last resort Key: IGNITE-9586 URL: https://issues.apache.org/jira/browse/IGNITE-9586 Project: Ignite Issue Type: Improvement Reporter: Ilya Kasnacheev As far as my understanding goes, it is a Docker gateway address. Nodes see that they have this address in common, and try to use it for Communication, which leads to confusing results since 172.17.0.1 is not actually shared between them. I think we should regard it as localhost or outright ignore it when picking address to connect to in Communication. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4749: IGNITE-9022: Changed class label output in SVM
GitHub user zaleslaw opened a pull request: https://github.com/apache/ignite/pull/4749 IGNITE-9022: Changed class label output in SVM You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9022 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4749.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4749 commit e1dd11ec45d088c42c1a4d3025f0b3e7f9b3ce73 Author: zaleslaw Date: 2018-09-13T15:14:27Z IGNITE-9022: Changed class label output in SVM ---
Re: Table Names in Spark Catalog
Hello, Stuart.
Do you need any assistance with this task from me or other community member?
В Вт, 04/09/2018 в 19:03 +0300, Nikolay Izhikov пишет:
> Hello, Stuart.
>
> Sorry for the silence.
>
> I was swamped the last couple of days.
>
> I think you can go forward and implement suggested solution.
> I'm -0 with it.
> So no block from my side, but I'm still no happy with abstractions :).
>
> В Пн, 03/09/2018 в 09:35 +0100, Stuart Macdonald пишет:
> > Nikolay, Val, it would be good if we could reach agreement here so that I
> > can make the necessary modifications before the 2.7 cutoff.
> >
> > Nikolay - would you be comfortable if I went ahead and made database=schema?
> >
> > Stuart.
> >
> > On Mon, Aug 27, 2018 at 10:22 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Hi Nikolay,
> > >
> > > I think it's actually pretty unfortunate that Spark uses term "database"
> > > here, as it essentially refers to a schema in my view. Usually, database
> > > is
> > > something you create a physical connection to, and connection is bind to
> > > that database. To connect to another database you need to create a new
> > > connection. In Spark, however, you can switch between "databases" within a
> > > single session, which looks really weird to me because it's usually a
> > > characteristic of a schema. Having said that, I understand your concern,
> > > but I don't think there is an ideal solution.
> > >
> > > As for your approach, I still don't understand how it will allow to fully
> > > support schemas in catalog.
> > > - How will you get a list of tables within a particular schema? In other
> > > words, what would listTables() method return?
> > > - How will you switch between the schemas?
> > > - Etc.
> > >
> > > I still think assuming database=schema is the best we can do here, but I
> > > would be happy to hear another opinions from other community members.
> > >
> > > OPTION_SCHEMA should definitely be introduced though (I thought we already
> > > did, no?). CREATE TABLE will be supported with this ticket:
> > > https://issues.apache.org/jira/browse/IGNITE-5780. For now we will have to
> > > throw an exception if custom schema name is provided when creating a Spark
> > > session, but table does not exist yet.
> > >
> > > -Val
> > >
> > > On Sun, Aug 26, 2018 at 7:56 AM Nikolay Izhikov
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > Personally, I don't like the solution with database == schema name.
> > > >
> > > > 1. I think we should try to use the right abstractions.
> > > > schema == database doesn't sound right for me.
> > > >
> > > > Do you want to answer to all of our users something like that:
> > > >
> > > > - "How I can change Ignite SQL schema?"
> > > > - "This is obvious, just use setDatabase("MY_SCHEMA_NAME")".
> > > >
> > > > 2. I think we restrict whole solution with that decision.
> > > > If Ignite will support multiple databases in the future we just don't
> > >
> > > have
> > > > a place for it.
> > > >
> > > > I think we should do the following:
> > > >
> > > > 1. IgniteExternalCatalog should be able to return *ALL* tables
> > > > within Ignite instance.
> > > > We shouldn't restrict tables list by schema by default.
> > > > We should return tables with schema name - `schema.table`
> > > >
> > > > 2. We should introduce `OPTION_SCHEMA` for a dataframe to
> > > > specify
> > > > a schema.
> > > >
> > > > There is an issue with the second step: We can't use schema name
> > > > in `CREATE TABLE` clause.
> > > > This is restriction of current Ignite SQL.
> > > >
> > > > I propose to make the following:
> > > >
> > > > 1. For all write modes that requires the creation of table we
> > > > should disallow usage of table outside of `SQL_PUBLIC`
> > > > or usage of `OPTION_SCHEMA`. We should throw proper exception
> > > > for
> > > > this case.
> > > >
> > > > 2. Create a ticket to support `CREATE TABLE` with custom schema
> > > > name.
> > > >
> > > > 3. After resolving ticket from step 2 we can add full support of
> > > > custom schema to Spark integration.
> > > >
> > > > 4. We should throw an exception if user try to use setDatabase.
> > > >
> > > > Is that makes sense for you?
> > > >
> > > > В Вс, 26/08/2018 в 14:09 +0100, Stuart Macdonald пишет:
> > > > > I'll go ahead and make the changes to represent the schema name as the
> > > > > database name for the purposes of the Spark catalog.
> > > > >
> > > > > If anyone knows of an existing way to list all available schemata
> > >
> > > within
> > > > an
> > > > > Ignite instance please let me know, otherwise the first task will be
> > > > > creating that mechanism.
> > > > >
> > > > > Stuart.
> > > > >
> > > > > On Fri, Aug 24, 2018 at 6:23 PM Valentin Kulichenko <
> > > > > valentin.kuliche...@gmail.com> wrote:
> > > > >
> > > > > > Nikolay,
> > > > > >
> > > > > > If there are
[jira] [Created] (IGNITE-9585) Error message sometimes refers nonexisting log file when remote node fails to start
Oleg Ignatenko created IGNITE-9585:
--
Summary: Error message sometimes refers nonexisting log file when
remote node fails to start
Key: IGNITE-9585
URL: https://issues.apache.org/jira/browse/IGNITE-9585
Project: Ignite
Issue Type: Bug
Affects Versions: 2.6
Reporter: Oleg Ignatenko
Teamcity build logs sometimes refer to remote node log files that aren't
present in build artifacts
([example|https://ci.ignite.apache.org/viewLog.html?buildTypeId=IgniteTests24Java8_StartNodes=1849937_IgniteTests24Java8_StartNodes=%3Cdefault%3E]).
I managed to reproduce this on my machine (details below) and it looks like
typically the root cause of this is error message from
[StartNodeCallableImpl|https://github.com/apache/ignite/blob/master/modules/ssh/src/main/java/org/apache/ignite/internal/util/nodestart/StartNodeCallableImpl.java]
referring readers to file that doesn't exist (and it wasn't even created to
start with).
{code:java}
return new ClusterStartNodeResultImpl(spec.host(), false, "Remote
node could not start. " +
"See log for details: " + scriptOutputPath);
{code}
This is quite painful when one tries to investigate node launching failures
because the misleading message causes one to waste time investigating the
problem that doesn't exist (it appears as if log file was there but somehow
disappeared for some mysterious reason).
To reproduce the issue locally one can do as follows: first, modify file
[start-nodes-custom.sh|https://github.com/apache/ignite/blob/master/modules/core/src/test/bin/start-nodes-custom.sh]
by replacing {{ignite.sh}} with the name of script that doesn't exist, eg
{{noignite.sh}}. After that, execute unit test
[IgniteProjectionStartStopRestartSelfTest|https://github.com/apache/ignite/blob/master/modules/ssh/src/test/java/org/apache/ignite/internal/IgniteProjectionStartStopRestartSelfTest.java]
and study its results and logs.
You will find that {{testCustomScript}} fails - which is expected because name
of the script intended to be executed has changed to one that doesn't exist.
Also you will find that log file for respective node hasn't been created -
which is also expected because shell command fails before creating it. But in
the same time test log will refer to mentioned file as if it exists.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ignite pull request #4748: Ignite 2.5.1 master
GitHub user macrergate opened a pull request: https://github.com/apache/ignite/pull/4748 Ignite 2.5.1 master You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-2.5.1-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4748.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4748 commit 0ad44bc27bc79630b03748b060aa2503e0830d31 Author: Alexey Goncharuk Date: 2018-04-24T10:26:12Z IGNITE-8369 Fixed ZkCommunicationErrorProcessFuture to use j.u.c.ConcurrentHashMap commit 198d6d2a6c66100a38212373e61014e746b364e3 Author: Pavel Kovalenko Date: 2018-04-24T13:08:21Z IGNITE-8339 Fixed partition state detection on activation - Fixes #3885. Signed-off-by: Alexey Goncharuk commit b4bf53f81a4ed06b61ca207ce33ba5e859e6ce40 Author: Alexey Goncharuk Date: 2018-04-24T17:57:56Z IGNITE-8082 Fixed build (added javadoc) commit b96445247a547e538d923f6898b85bde1b8b17b9 Author: Alexey Goncharuk Date: 2018-04-25T09:30:09Z ignite-2.5.1 Fixed version for backward compatibility commit f6cb457a713bd482b202f98a795f711aa9243638 Author: Alexey Goncharuk Date: 2018-04-25T11:57:15Z ignite-2.5.1 Fixed version for backward compatibility commit 6c87080156c39f7dd70692df2f46eff8b9846a8e Author: Pavel Kovalenko Date: 2018-04-25T14:39:02Z IGNITE-8358 Destroy partition inside evictor to prevent possible deadlock. This closes #3911. commit 3b93a70025f0295f1d016b981603592d3ba7a404 Author: Aleksey Plekhanov Date: 2018-04-25T15:40:46Z IGNITE-8374 Fixed hanging of state transition completeion on cluster().activate() Signed-off-by: Andrey Gura commit bae07ead40dea7f1a44688f0567f265ca6b20294 Author: Sergey Chugunov Date: 2018-04-25T17:59:25Z IGNITE-8372 ZookeeperClusterNode was made Externalizable to preserve serializing local node's metrics - Fixes #3907. Signed-off-by: Alexey Goncharuk commit 3f62a454a27588b49983c6e00a17568d7a2db634 Author: Pavel Kovalenko Date: 2018-04-26T13:06:52Z IGNITE-8390 Correct assertion for historical rebalance - Fixes #3917. Signed-off-by: Alexey Goncharuk commit 2f2ad0fd35c2928fb117ad8f0b0b5e05ff8a9074 Author: Ivan Rakov Date: 2018-04-26T15:57:38Z IGNITE-8277 Added utilities to check and display cache info commit 19fd136338a6571ae352df424fa7a06e8c537312 Author: Alexey Goncharuk Date: 2018-04-26T16:38:03Z IGNITE-8404 Fixed NPE in MappedFileMemoryProvider commit 1676560a844402348ee4882e8c9f9d798866697e Author: Alexey Goncharuk Date: 2018-04-27T10:08:03Z IGNITE-8408 Fixed IgniteUtils.invoke commit e10baa510c8855bbe6344c24b4f8c6db5fc086f4 Author: Evgeny Stanilovskiy Date: 2018-04-25T17:46:47Z IGNITE-8066 AssertionError while trying to archive wal segment. - Fixes #3837. Signed-off-by: dpavlov (cherry picked from commit 1fb2369) commit 48601cbd224e89e8396e53b4406f7a65e6fc6d9d Author: Ivan Rakov Date: 2018-04-27T12:06:08Z IGNITE-8393 Unexpected error during WAL compression fixed Signed-off-by: Andrey Gura commit 23e802e5b8146677a8e05767d3db93991fb0831a Author: Ivan Rakov Date: 2018-04-27T19:15:15Z IGNITE-8416 CommandHandlerParsingTest stably fails with parsing error Signed-off-by: Andrey Gura commit 7412288aa5f00533b2b1ece07b245a0029242add Author: Ivan Rakov Date: 2018-05-08T12:03:00Z IGNITE-8429 Unexpected error during incorrect WAL segment decompression, causes node termination Signed-off-by: Andrey Gura (cherry picked from commit f3a61e4) (cherry picked from commit 104b771) commit 6c44ebc51b31d04f392b7248ea479bfd12af06cd Author: Ivan Rakov Date: 2018-05-08T12:49:59Z IGNITE-8429 Unexpected error during incorrect WAL segment decompression, causes node termination + synchronized block (cherry picked from commit 99a07ed) commit 561e7fafa34b638b95f6007c259436a6183f0c47 Author: Anton Kalashnikov Date: 2018-05-08T13:35:05Z IGNITE-8435 StorageException is handled like NodeStoppingException during failing transaction commit Signed-off-by: Andrey Gura (cherry picked from commit e8b846b) commit b3ca15c36765652d311502f9e2d0ddb83ef4fc9d Author: Anton Kalashnikov Date: 2018-05-08T14:10:53Z IGNITE-8430 implemented getCurrentCoordinatorFormatted method in IgniteMXBean - Fixes #3957. Signed-off-by: dpavlov (cherry picked from commit 666f220) commit 383c0de3c92003650c59b20246f82df89f098c79 Author: ascherbakoff Date: 2018-05-10T09:08:17Z IGNITE-8437: Control utility fails to connect to cluster if zookeeper discovery used. This closes #3950. (cherry picked from commit 3cdee53) commit 0fa7e8de720c813a2152dfa7526f5234a658e01a Author: Pavel
[jira] [Created] (IGNITE-9584) .NET DataStorageMetricsTest is flaky in master
Alexey Goncharuk created IGNITE-9584: Summary: .NET DataStorageMetricsTest is flaky in master Key: IGNITE-9584 URL: https://issues.apache.org/jira/browse/IGNITE-9584 Project: Ignite Issue Type: Test Reporter: Alexey Goncharuk -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: PHP thin client
Good. 5) I've filed a separate ticket, so it can be taken by a someone, who is familiar with our build system: [1] [1] - https://issues.apache.org/jira/browse/IGNITE-9465 Best Regards, Igor On Thu, Sep 13, 2018 at 5:07 PM Alexey Kosenchuk < alexey.kosenc...@nobitlost.com> wrote: > 11.09.2018 14:46, Igor Sapego пишет: > > 1) I meant auto-generated doc, this page: [1]. By the way, > > many methods do not have any description as well. > > Updated - introduction sentences placed on the main page of the > autogenerated API spec. > All public methods have description, except setters for cache > configuration which do not need it. > > > 2) You can, for example, print object as you generate and > > then get them from cache. > > Done. > > > 3) Yes, 10800 should be the default port. > > Done. > > All changes are in the updated PR. > > > 4) You are right, I've just used server with SSL enabled. > > Everything works now. > > Good. > > > 5) Ok, can you just provide a list of directories/files, that > > should be copied to binary release? > > ignite/modules/platforms/php/composer.json > ignite/modules/platforms/php/src > ignite/modules/platforms/php/examples > > If cfg file to generate the API spec should be in the release > ignite/modules/platforms/php/api_docs/Doxyfile > > If tests should be in the release > ignite/modules/platforms/php/tests > > Regards, > -Alexey > > > > > [1] - > > > https://rawgit.com/nobitlost/ignite/ignite-7783-docs/modules/platforms/php/api_docs/html/index.html > > > > Best Regards, > > Igor > > > > > > On Tue, Sep 11, 2018 at 1:31 AM Alexey Kosenchuk < > > alexey.kosenc...@nobitlost.com> wrote: > > > >> Hi Igor, > >> > >> thanks for the review. > >> > >> Pls see below... > >> > >> >> 1. Main page for documentation is empty. > >> > >> What is the main page for documentation? > >> > >> As wrote, > >> the auto-generated API spec is here: > >> > >> > https://rawgit.com/nobitlost/ignite/ignite-7783-docs/modules/platforms/php/api_docs/html/index.html > >> > >> All other docs are here: > >> > >> > https://github.com/nobitlost/ignite/blob/ignite-7783-docs/modules/platforms/php/README.md > >> Going to be placed on readme.io by Prachi (thanks!) > >> https://issues.apache.org/jira/browse/IGNITE-9523 > >> > >> >> 2. More output for Auth example is needed. This is not a test, > after > >> all, but example. > >> > >> Will add a log when a connection happens with the details of the > >> connection. Nothing else to output in this trivial example. > >> In fact, the real profit for a user here is the source code with an > >> example of TLS/auth cfg for the client. > >> > >> >> 3. If I try run test with APACHE_IGNITE_CLIENT_ENDPOINTS=127.0.0.1, > >> they fail > >> > >> Do you mean 10800 should be the default port when not specified by a > user? > >> Will add. > >> > >> >> 4. If I try run test with > >> APACHE_IGNITE_CLIENT_ENDPOINTS=127.0.0.1:10800, they fail > >> > >> Double checked exactly in the same environment you have. It works. > >> > >> Please try again. > >> Do you use a server with all default settings? Did not you try with the > >> server after the Auth example? > >> Do other examples work with the same server? > >> If you still see the problem with the tests, please send the server's > >> log to us. And/or switch the client's debug on - call setDebug(true) - > >> and share the client's output. > >> > >> >> 5. When "maven package" command is executed on Ignite, no php > >> directory appears > >> > >> Need a help from experts / release engineer. > >> > >> Thanks, > >> -Alexey > >> > >> 10.09.2018 15:34, Igor Sapego пишет: > >>> By the way, I used Ubuntu 18.04, PHP 7.2.7 and what seems > >>> to be PhpUnit 7.3 (not sure here). > >>> > >>> Best Regards, > >>> Igor > >>> > >>> > >>> On Mon, Sep 10, 2018 at 3:28 PM Igor Sapego > wrote: > >>> > Guys, I've reviewed the API (which looks good), run tests and examples > >> and > here are my commments: > > 1. Main page for documentation is empty. > > 2. More output for Auth example is needed. This is not a test, after > >> all, > but example. > > 3. If I try run test with APACHE_IGNITE_CLIENT_ENDPOINTS=127.0.0.1, > they > fail with the following message: > Apache\Ignite\Exception\NoConnectionException: [127.0.0.1] Failed to > >> parse > address "127.0.0.1" in > > >> > /home/isapego/work/ignite/modules/platforms/php/src/Apache/Ignite/Internal/Connection/ClientFailoverSocket.php:107 > Stack trace: > #0 > > >> > /home/isapego/work/ignite/modules/platforms/php/src/Apache/Ignite/Internal/Connection/ClientFailoverSocket.php(54): > > >> > Apache\Ignite\Internal\Connection\ClientFailoverSocket->failoverConnect() > #1 > > >> > /home/isapego/work/ignite/modules/platforms/php/src/Apache/Ignite/Client.php(61): > > >> > Apache\Ignite\Internal\Connection\ClientFailoverSocket->connect(Object(Apache\Ignite\ClientConfiguration)) > #2 >
[jira] [Created] (IGNITE-9583) PHP thin: php directory should be included in binary release
Igor Sapego created IGNITE-9583:
---
Summary: PHP thin: php directory should be included in binary
release
Key: IGNITE-9583
URL: https://issues.apache.org/jira/browse/IGNITE-9583
Project: Ignite
Issue Type: Improvement
Components: build, thin client
Reporter: Igor Sapego
Fix For: 2.7
Currently, php directory is not generated by the {{maven install}} command.
Need to add appropriate copy steps to {{assembly/release-fabric-base.xml}} file.
The following components should be copied:
{noformat}
ignite/modules/platforms/php/composer.json
ignite/modules/platforms/php/src
ignite/modules/platforms/php/examples
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: PHP thin client
11.09.2018 14:46, Igor Sapego пишет: 1) I meant auto-generated doc, this page: [1]. By the way, many methods do not have any description as well. Updated - introduction sentences placed on the main page of the autogenerated API spec. All public methods have description, except setters for cache configuration which do not need it. 2) You can, for example, print object as you generate and then get them from cache. Done. 3) Yes, 10800 should be the default port. Done. All changes are in the updated PR. 4) You are right, I've just used server with SSL enabled. Everything works now. Good. 5) Ok, can you just provide a list of directories/files, that should be copied to binary release? ignite/modules/platforms/php/composer.json ignite/modules/platforms/php/src ignite/modules/platforms/php/examples If cfg file to generate the API spec should be in the release ignite/modules/platforms/php/api_docs/Doxyfile If tests should be in the release ignite/modules/platforms/php/tests Regards, -Alexey [1] - https://rawgit.com/nobitlost/ignite/ignite-7783-docs/modules/platforms/php/api_docs/html/index.html Best Regards, Igor On Tue, Sep 11, 2018 at 1:31 AM Alexey Kosenchuk < alexey.kosenc...@nobitlost.com> wrote: Hi Igor, thanks for the review. Pls see below... >> 1. Main page for documentation is empty. What is the main page for documentation? As wrote, the auto-generated API spec is here: https://rawgit.com/nobitlost/ignite/ignite-7783-docs/modules/platforms/php/api_docs/html/index.html All other docs are here: https://github.com/nobitlost/ignite/blob/ignite-7783-docs/modules/platforms/php/README.md Going to be placed on readme.io by Prachi (thanks!) https://issues.apache.org/jira/browse/IGNITE-9523 >> 2. More output for Auth example is needed. This is not a test, after all, but example. Will add a log when a connection happens with the details of the connection. Nothing else to output in this trivial example. In fact, the real profit for a user here is the source code with an example of TLS/auth cfg for the client. >> 3. If I try run test with APACHE_IGNITE_CLIENT_ENDPOINTS=127.0.0.1, they fail Do you mean 10800 should be the default port when not specified by a user? Will add. >> 4. If I try run test with APACHE_IGNITE_CLIENT_ENDPOINTS=127.0.0.1:10800, they fail Double checked exactly in the same environment you have. It works. Please try again. Do you use a server with all default settings? Did not you try with the server after the Auth example? Do other examples work with the same server? If you still see the problem with the tests, please send the server's log to us. And/or switch the client's debug on - call setDebug(true) - and share the client's output. >> 5. When "maven package" command is executed on Ignite, no php directory appears Need a help from experts / release engineer. Thanks, -Alexey 10.09.2018 15:34, Igor Sapego пишет: By the way, I used Ubuntu 18.04, PHP 7.2.7 and what seems to be PhpUnit 7.3 (not sure here). Best Regards, Igor On Mon, Sep 10, 2018 at 3:28 PM Igor Sapego wrote: Guys, I've reviewed the API (which looks good), run tests and examples and here are my commments: 1. Main page for documentation is empty. 2. More output for Auth example is needed. This is not a test, after all, but example. 3. If I try run test with APACHE_IGNITE_CLIENT_ENDPOINTS=127.0.0.1, they fail with the following message: Apache\Ignite\Exception\NoConnectionException: [127.0.0.1] Failed to parse address "127.0.0.1" in /home/isapego/work/ignite/modules/platforms/php/src/Apache/Ignite/Internal/Connection/ClientFailoverSocket.php:107 Stack trace: #0 /home/isapego/work/ignite/modules/platforms/php/src/Apache/Ignite/Internal/Connection/ClientFailoverSocket.php(54): Apache\Ignite\Internal\Connection\ClientFailoverSocket->failoverConnect() #1 /home/isapego/work/ignite/modules/platforms/php/src/Apache/Ignite/Client.php(61): Apache\Ignite\Internal\Connection\ClientFailoverSocket->connect(Object(Apache\Ignite\ClientConfiguration)) #2 /home/isapego/work/ignite/modules/platforms/php/tests/TestingHelper.php(52): Apache\Ignite\Client->connect(Object(Apache\Ignite\ClientConfiguration)) #3 /home/isapego/work/ignite/modules/platforms/php/tests/SqlQueryTest.php(49): Apache\Ignite\Tests\TestingHelper::init() #4 /home/isapego/work/ignite/modules/platforms/php/vendor/phpunit/phpunit/src/Framework/TestSuite.php(703): Apache\Ignite\Tests\SqlQueryTestCase::setUpBeforeClass() #5 /home/isapego/work/ignite/modules/platforms/php/vendor/phpunit/phpunit/src/Framework/TestSuite.php(750): PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult)) #6 /home/isapego/work/ignite/modules/platforms/php/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(587): PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult)) #7
Re: Class field ThreadLocal. Why not static?
Dmitry, Thanks for your example! The approach looks similar to "problematic" on used in jdk described in [1]. Fortunately, in our case buffer is quite limited. On the other hand we always should be cautious because Ignite code frequently is running on user threads (e.g. it is typical for SQL). [1] http://www.evanjones.ca/java-bytebuffer-leak.html 2018-09-13 9:28 GMT+03:00 Alexey Goncharuk : > Maxim, > > If multiple instances of Ignite is started in the same JVM and a user > thread will access first one instance of Ignite, then another, you will end > up with the static thread local holding the last WAL pointer from the > second grid. This is possible, for example, when a user thread commits a > transaction or runs an atomic update on a data node. Any access of the > first Ignite instance will have an invalid thread-local value. > > вт, 11 сент. 2018 г. в 13:29, Maxim Muzafarov : > > > Alexey, Ivan, > > > > Agree. Keeping strong references to the Thread object is the source of > > memory leak with ThreadLocals variables > > and the values that it stores. ThreadLocalMap is bound to the Thread > > lifespan [1], so I think when we are using > > everything right all will be GC'ed correctly. > > Is this memory leaks with ThreadLocal's you mean, Alexey? If not, please, > > share your example. > > > > Also, agree that these usages should be bound to the component lifespan. > > But for `FileWriteAheadLogManager` > > I think this variable used not semantically right. I've dumped all > threads > > (total ~49 threads) > > that are using `lastWalPtr` in `FileWriteAheadLogManager`. For instance, > > * exchange-worker-#40%wal.IgniteWalRecoveryTest0% > > * sys-#148%wal.IgniteWalRecoveryTest1% > > * db-checkpoint-thread-#129%wal.IgniteWalRecoveryTest2% > > Suppose everything would be OK here for `static` and `non-static` case of > > ThreadLocal. > > > > [1] > > > > http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/ > share/classes/java/lang/Thread.java#l760 > > > > On Tue, 11 Sep 2018 at 13:05 Павлухин Иван wrote: > > > > > Dmitriy, > > > > > > Could you point to some piece of code implementing described pattern? > > > > > > 2018-09-11 13:02 GMT+03:00 Павлухин Иван : > > > > > > > Alex, > > > > > > > > ThreadLocal subclass is used in IgniteH2Indexing for simple access to > > H2 > > > > Connection from current thread. Such subclass has a capability to > > create > > > > connection if one does not exist, so obtaining connection is merely > > > > ThreadLocal.get. Also there are scheduled routines to cleanup > > connections > > > > and associated with them statement cache after some expiration time. > > For > > > > that reason Map is maintained. As query > > can > > > > run on user thread we need to cleanup mentioned map to avoid a leak > > when > > > > Thread is terminated. So we need to check thread status in cleanup > > > routines > > > > and remove entries for terminated Threads. And historically there was > > no > > > > cleanup for terminated threads and leak was possible. And also great > > care > > > > must be taken in order to avoid cyclic reference between ThreadLocal > > > > instance and a stored value. Which easily could occur if the stored > > value > > > > is covered by multiple layers of abstraction. > > > > > > > > And I am describing some historical state. Now machinery in > > > IgniteH2Indexing > > > > is even more complex (I hope we will have a chance to improve it). > > > > > > > > 2018-09-11 11:00 GMT+03:00 Alexey Goncharuk < > > alexey.goncha...@gmail.com > > > >: > > > > > > > >> Ivan, > > > >> > > > >> Can you elaborate on the issue with the thread local cleanup you've > > > faced? > > > >> > > > >> вт, 11 сент. 2018 г. в 9:13, Павлухин Иван : > > > >> > > > >> > Guys, > > > >> > > > > >> > As we know ThreadLocal is an instrument which should be used with > > > great > > > >> > care. And I recently faced with problems related to proper cleanup > > of > > > >> > ThreadLocal which is not needed anymore. In my opinion the best > > thing > > > >> (in > > > >> > ideal world) is to get rid of ThreadLocal where possible, but I > > guess > > > >> that > > > >> > it is quite hard (in real world). > > > >> > > > > >> > Also, a question comes to my mind. As ThreadLocal is so common in > > our > > > >> code, > > > >> > could you suggest some guidance or code fragments which address > > proper > > > >> > ThreadLocal > > > >> > lifecycle control and especially cleanup? > > > >> > > > > >> > 2018-09-10 12:46 GMT+03:00 Alexey Goncharuk < > > > alexey.goncha...@gmail.com > > > >> >: > > > >> > > > > >> > > Maxim, > > > >> > > > > > >> > > Ignite supports starting multiple instances of Ignite in the > same > > > VM, > > > >> so > > > >> > > having static thread locals for the fields you mentioned does > not > > > >> work. > > > >> > > > > > >> > > Generally, I think thread-local should be bound to the lifespan > of > > > the > > > >> > > component it describes. Static thread-locals are hard to > clean-up > >
[GitHub] asfgit closed pull request #8: Trigger build bugfix
asfgit closed pull request #8: Trigger build bugfix
URL: https://github.com/apache/ignite-teamcity-bot/pull/8
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/util/HttpUtil.java
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/util/HttpUtil.java
index 1184869..4730606 100644
--- a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/util/HttpUtil.java
+++ b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/util/HttpUtil.java
@@ -93,8 +93,10 @@ public static InputStream sendGetToGit(String githubAuthTok,
String url) throws
URL obj = new URL(url);
HttpURLConnection con = (HttpURLConnection)obj.openConnection();
+if (githubAuthTok != null)
+con.setRequestProperty("Authorization", "token " + githubAuthTok);
+
con.setRequestProperty("accept-charset",
StandardCharsets.UTF_8.toString());
-con.setRequestProperty("Authorization", "token " + githubAuthTok);
con.setRequestProperty("Connection", "Keep-Alive");
con.setRequestProperty("Keep-Alive", "header");
diff --git
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
index 2148a35..9269cbe 100644
---
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
+++
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/TriggerBuild.java
@@ -32,6 +32,7 @@
import org.apache.ignite.ci.github.PullRequest;
import org.apache.ignite.ci.tcmodel.result.Build;
import org.apache.ignite.ci.user.ICredentialsProv;
+import org.apache.ignite.ci.util.XmlUtil;
import org.apache.ignite.ci.web.CtxListener;
import org.apache.ignite.ci.web.rest.login.ServiceUnauthorizedException;
import org.apache.ignite.ci.web.model.SimpleResult;
@@ -56,6 +57,7 @@ public SimpleResult triggerBuild(
@Nullable @QueryParam("top") Boolean top,
@Nullable @QueryParam("observe") Boolean observe
) {
+String errors = "";
final ICredentialsProv prov = ICredentialsProv.get(req);
if (!prov.hasAccess(srvId))
@@ -64,33 +66,37 @@ public SimpleResult triggerBuild(
ITcHelper helper = CtxListener.getTcHelper(context);
try (final ITeamcity teamcity = helper.server(srvId, prov)) {
-PullRequest pr = teamcity.getPullRequest(branchName);
-
-String ticketId = "";
+Build build = teamcity.triggerBuild(suiteId, branchName, false,
top != null && top);
-if (pr.getTitle().startsWith("IGNITE-")) {
-int beginIdx = 7;
-int endIdx = 7;
+if (observe != null && observe) {
+PullRequest pr = teamcity.getPullRequest(branchName);
-while (endIdx < pr.getTitle().length() &&
Character.isDigit(pr.getTitle().charAt(endIdx)))
-endIdx++;
+String ticketId = "";
-ticketId = pr.getTitle().substring(beginIdx, endIdx);
-}
+if (pr.getTitle().startsWith("IGNITE-")) {
+int beginIdx = 7;
+int endIdx = 7;
-if (ticketId.equals(""))
-return new SimpleResult("PR title \"" + pr.getTitle() + "\"
should starts with \"IGNITE-\"." +
-" Please, rename PR according to the" +
-" contributing
guide.");
+while (endIdx < pr.getTitle().length() &&
Character.isDigit(pr.getTitle().charAt(endIdx)))
+endIdx++;
-Build build = teamcity.triggerBuild(suiteId, branchName, false,
top != null && top);
+ticketId = pr.getTitle().substring(beginIdx, endIdx);
+}
-if (observe != null && observe)
-helper.buildObserver().observe(build, srvId, prov, "ignite-" +
ticketId);
+if (!ticketId.equals(""))
+helper.buildObserver().observe(build, srvId, prov,
"ignite-" + ticketId);
+else {
+errors += "" +
+"JIRA ticket will not be notified after the tests are
completed - " +
+"PR title \"" + pr.getTitle() + "\" should starts with
\"IGNITE-\"." +
+" Please, rename PR according to the" +
+" contributing guide.";
+}
+}
}
-return new SimpleResult("OK");
+return new SimpleResult("Tests started." + errors);
}
@GET
diff --git a/ignite-tc-helper-web/src/main/webapp/index.html
Re: Workflow improvement
Hi, Dmitriy, Can you give me rights to stop my builds on TeamCity? Working on the TCH, I run a lot of builds, and I don't like to ask other people to stop builds too often. 2018-09-10 23:53 GMT+03:00 Dmitrii Ryabov : > Hi, Dmitriy, > > I made PRs in my fork for test purposes. Real PRs were made to the GitHub > mirror, and one of them is already merged by D. Govorukhin. PR with GitHub > statuses [1] is ready for review. PR with JIRA comment will be ready in a > few days. > > [1] https://github.com/apache/ignite-teamcity-bot/pull/5 > > 2018-09-10 23:00 GMT+03:00 Dmitriy Pavlov : > >> Hi Dmitrii, >> >> I deployed change with blockers summary of failures at the top of PR >> result >> page. The Bot is migrating entries now, I hope it will be done in 1-4 >> hours. >> >> I noticed your PRs are created in your own fork, not in >> https://github.com/apache/ignite-teamcity-bot >> Could you please create unmerged PR in GitHub mirror so it could be merged >> after reviewing by the apply-pr script. >> >> Sincerely, >> Dmitriy Pavlov >> >> пн, 3 сент. 2018 г. в 18:20, Dmitrii Ryabov : >> >> > Hi, Dmitriy, >> > >> > I created a table with possible blockers [1] for the page with the >> latest >> > build analysis. Anton K. has reviewed it. Can you check and merge it? >> > >> > Also, I created a button to notify PR about analisis results [2]. I used >> > GitHub statuses (example [3]), but to work it needs a token with push >> > permission. As Apache PMC, can you acquire such token? I think statuses >> are >> > much more notable than comments. >> > >> > [1] https://issues.apache.org/jira/browse/IGNITE-9376 >> > [2] https://issues.apache.org/jira/browse/IGNITE-9377 >> > [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls >> > >> > 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov : >> > >> > > Not sure I understood what bot statistic reduction means. >> > > >> > > As far as I understood, you also mean locating failure automatically >> with >> > > re-runs on specific git revisions (hashes). I found it will be a good >> > > contribution to TC Bot for both flaky and non-flaky failures >> detection. >> > E.g >> > > failure occurred after 10 commits merged to master. How to find out >> bad >> > > commit? Automatic location (analog of bisecting, but using TC test) >> will >> > be >> > > really helpful. But it is not a simple contribution. >> > > >> > > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov : >> > > >> > > > Hi, Dmitriy, >> > > > >> > > > Good, I'll create tickets for this steps. >> > > > >> > > > Stable suites are a good idea, but also we need to do some actions >> > when a >> > > > flaky test will appear in stable suite of master branch run. To find >> > out >> > > a >> > > > guilty commit, we should run previous commits on empty TeamCity's >> > queue. >> > > > This runs will reduce bot's statistics for the latest commits. >> > > > >> > > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov : >> > > > >> > > > > Hi Dmitriy, >> > > > > >> > > > > Sounds like a plan ;) I totally agree. >> > > > > Just one proposal: I would like to avoid hiding any test >> failures. 2 >> > > > > separate tables named 'Possible Blockers' and 'Other failures' >> should >> > > be >> > > > > completely clear. Comment to PR can contain only first category. >> > > > > >> > > > > We had a private discussion with Anton K. and he proposed a very >> > > > > interesting thing, I would like to bring it here. We can add >> > > > configuration >> > > > > into TC bot and mark some tests and some suites as 'Known Stable' >> It >> > > > means >> > > > > that suite or test failure should be considered as a blocker for >> > merge >> > > > > every time it fails, even if fail rate is non-zero. Very first >> list >> > of >> > > > such >> > > > > suites are >> > > > > - Build Apache Ignite >> > > > > - License check >> > > > > And tests: >> > > > > - .Net API Parity check >> > > > > Please share your vision. >> > > > > >> > > > > Meanwhile, I've updated the TC bot. >> > > > > - Underlying Apache Ignite DB was refactored to use lower >> partitions >> > > > count, >> > > > > so restart should be faster. >> > > > > - Now 100 builds are saved as our failure rate statistics basis. >> > > > Flakiness >> > > > > border was not changed, so more test now will be considered as >> flaky. >> > > > > - Now 4 builds required to be failed or timed out in a row before >> > > > > notification is sent. >> > > > > >> > > > > Sincerely, >> > > > > Dmitriy Pavlov >> > > > > >> > > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov < >> somefire...@gmail.com>: >> > > > > >> > > > > > Hi, Dmitriy, >> > > > > > >> > > > > > I propose the next steps: >> > > > > > >> > > > > > 1. Show current 0% tests in a separate table at the top of the >> > > analysis >> > > > > > results page. Thus, we'll see most suspicious (new or very >> flaky) >> > > tests >> > > > > > firstly. Or we can hide other tests under "More >>" button, like >> > top >> > > > long >> > > > > > running tests. >> > > > > > 2. Create a
[jira] [Created] (IGNITE-9582) Document Model Updating
Aleksey Zinoviev created IGNITE-9582: Summary: Document Model Updating Key: IGNITE-9582 URL: https://issues.apache.org/jira/browse/IGNITE-9582 Project: Ignite Issue Type: Task Components: documentation, ml Reporter: Aleksey Zinoviev Assignee: Alexey Platonov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: IoC/DI support in Apache Ignite.NEt
Hello,
I propose the following.
*Development*
1. Define *IDependencyResolver* in *Apache.Ignite.Core* under
*DependencyInjection* folder that contains at least 2 methods:
*T Resolve(object arguments = null);*
*object Resolve(Type type, object arguments = null);*
2. Create *InjectResolvableAttribute* in Apache.Ignite.Core under
*DependencyInjection* folder marked with
*[AttributeUsage(AttributeTargets.Field
| AttributeTargets.Property)]*
3. Create internal static class *DependencyInjector* with at least one
internal static method that will inject required dependencies in service
implementations like this one (exception checking is omitted currently and
has to be added)
*namespace Apache.Ignite.Core.DependencyInjection*
*{*
*using System.Linq;*
*using System.Reflection;*
*using Apache.Ignite.Core.Services;*
*internal static class DependencyInjector*
*{*
*internal static void Inject(IService svc, IIgnite ignite)*
*{*
*var dependencyResolver = ignite.GetDependencyResolver();*
*if (dependencyResolver == null)*
*{*
*return;*
*}*
*foreach (var property in svc.GetType()*
*.GetProperties(BindingFlags.Public |
BindingFlags.NonPublic | BindingFlags.Instance))*
*{*
*if
(!property.GetCustomAttributes(typeof(InjectResolvableAttribute),
true).Any())*
*{*
*continue;*
*}*
*if (!property.CanWrite)*
*{*
*continue;*
*}*
*var resolvedDependency =
dependencyResolver.Resolve(property.PropertyType);*
*property.SetValue(svc, resolvedDependency, null);*
*}*
*foreach (var field in svc.GetType()*
*.GetFields(BindingFlags.Public | BindingFlags.NonPublic |
BindingFlags.Instance))*
*{*
*if
(!field.GetCustomAttributes(typeof(InjectResolvableAttribute), true).Any())*
*{*
*continue;*
*}*
*var resolvedDependency =
dependencyResolver.Resolve(field.FieldType);*
*field.SetValue(svc, resolvedDependency);*
*}*
*}*
*}*
*}*
4. Add* DependencyInjector.Inject(svc, _ignite);* call to
*UnmanagedCallbacks.cs* right after *ResourceProcessor.Inject(svc, _ignite)*
.
5. Add 2 methods to *Ignite* class and it's interface *IIgnite*:
*public IDependencyResolver GetDependencyResolver();*
*public IIgnite SetDependencyResolver(IDependencyResolver
dependencyResolver);*
6. Create *Apache.Ignite.DependencyInjection.CastleWindsor* project and
implement *IDependencyResolver* with *CastleWindsorDependencyResolver*
class.
*namespace Apache.Ignite.DependencyInjection.CastleWindsor*
*{*
*using System;*
*using Castle.MicroKernel;*
*using IDependencyResolver =
Core.DependencyInjection.IDependencyResolver;*
*public class CastleWindsorDependencyResolver : IDependencyResolver*
*{*
*private IKernel Kernel { get; set; }*
*public CastleWindsorDependencyResolver(IKernel kernel)*
*{*
*Kernel = kernel;*
*}*
*public T Resolve(object arguments = null)*
*{*
*return arguments == null ? Kernel.Resolve() :
Kernel.Resolve(arguments);*
*}*
*public object Resolve(Type type, object arguments = null)*
*{*
*return arguments == null ? Kernel.Resolve(type) :
Kernel.Resolve(type, arguments);*
*}*
*}*
*}*
7. Create extension like this one
namespace Apache.Ignite.DependencyInjection.CastleWindsor
{
using Apache.Ignite.Core;
using Castle.MicroKernel;
public static class IgniteExtensions
{
public static IIgnite UseCastleWindsorResolver(this IIgnite ignite,
IKernel kernel)
{
return ignite.SetDependencyResolver(new
CastleWindsorDependencyResolver(kernel));
}
}
}
8. Write test.
*Usage*
var grid = Ignition.Start();
using (var container = new WindsorContainer())
{
container.Register(Component.For().ImplementedBy());
grid.UseCastleWindsorResolver(container.Kernel);
grid.GetServices().DeployClusterSingleton("FooService", new FooService());
}
What do you think?
Cheers,
Artyom.
On Mon, Sep 10, 2018 at 5:23 PM Pavel Tupitsyn wrote:
> Hi Artyom,
>
> Ticket already exists: https://issues.apache.org/jira/browse/IGNITE-9299
> Feel free to grab it.
>
> But before diving deep into coding, can you please provide a short overview
> of the proposed implementation?
> Either on dev list or in JIRA ticket directly.
>
> Thanks,
> Pavel
>
> On Mon, Sep 10, 2018 at 11:29 AM Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Hello Artyom,
> >
> > Welcome to the Apache Ignite community! I've added you to the list of
> > contributors, you should now be able to assign tickets to yourself.
> >
> > Get familiar with
[jira] [Created] (IGNITE-9581) Document ANN algorithm based on ACD concept
Aleksey Zinoviev created IGNITE-9581: Summary: Document ANN algorithm based on ACD concept Key: IGNITE-9581 URL: https://issues.apache.org/jira/browse/IGNITE-9581 Project: Ignite Issue Type: Task Components: documentation, ml Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev Fix For: 2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4747: IGNITE-9580: Added setMaxSize for ignite configur...
GitHub user avplatonov opened a pull request: https://github.com/apache/ignite/pull/4747 IGNITE-9580: Added setMaxSize for ignite configuration in test You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9580 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4747.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4747 commit d0a14f0de913a977363ed21bb3c8bf54b35b388a Author: Alexey Platonov Date: 2018-09-13T13:01:02Z added setMaxSize for ignite configuration in test ---
[jira] [Created] (IGNITE-9580) Fix exit code 137 in Query 1 Suite
Alexey Platonov created IGNITE-9580: --- Summary: Fix exit code 137 in Query 1 Suite Key: IGNITE-9580 URL: https://issues.apache.org/jira/browse/IGNITE-9580 Project: Ignite Issue Type: Bug Reporter: Alexey Platonov Assignee: Alexey Platonov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
The future of Affinity / Topology concepts and possible PME optimizations.
Igniters, As most of you know, Ignite has a concept of AffinityTopologyVersion, which is associated with nodes that are currently present in topology and a global cluster state (active/inactive, baseline topology, started caches). Modification of either of them involves process called Partition Map Exchange (PME) and results in new AffinityTopologyVersion. At that moment all new cache and compute grid operations are globally "frozen". This might lead to indeterminate cache downtimes. However, our recent changes (esp. introduction of Baseline Topology) caused me to re-think those concept. Currently there are many cases when we trigger PME, but it isn't necessary. For example, adding/removing client node or server node not in BLT should never cause partition map modifications. Those events modify the *topology*, but *affinity* in unaffected. On the other hand, there are events that affect only *affinity* - most straightforward example is CacheAffinityChange event, which is triggered after rebalance is finished to assign new primary/backup nodes. So the term *AffinityTopologyVersion* now looks weird - it tries to "merge" two entities that aren't always related. To me it makes sense to introduce separate *AffinityVersion *and *TopologyVersion*, review all events that currently modify AffinityTopologyVersion and split them into 3 categories: those that modify only AffinityVersion, only TopologyVersion and both. It will allow us to process such events using different mechanics and avoid redundant steps, and also reconsider mapping of operations - some will be mapped to topology, others - to affinity. Here is my view about how different event types theoretically can be optimized: 1. Client node start / stop: as stated above, no PME is needed, ticket https://issues.apache.org/jira/browse/IGNITE-9558 is already in progress. 2. Server node start / stop not from baseline: should be similar to the previous case, since nodes outside of baseline cannot be partition owners. 3. Start node in baseline: both affinity and topology versions should be incremented, but it might be possible to optimize PME for such case and avoid cluster-wide freeze. Partition assignments for such node are already calculated, so we can simply put them all into MOVING state. However, it might take significant effort to avoid race conditions and redesign our architecture. 4. Cache start / stop: starting or stopping one cache doesn't modify partition maps for other caches. It should be possible to change this procedure to skip PME and perform all necessary actions (compute affinity, start/stop cache contexts on each node) in background, but it looks like a very complex modification too. 5. Rebalance finish: it seems possible to design a "lightweight" PME for this case as well. If there were no node failures (and if there were, PME should be triggered and rebalance should be cancelled anyways) all partition states are already known by coordinator. Furthermore, no new MOVING or OWNING node for any partition is introduced, so all previous mappings should still be valid. For the latter complex cases in might be necessary to introduce "is compatible" relationship between affinity versions. Operation needs to be remapped only if new version isn't compatible with the previous one. Please share your thoughts. -- Best regards, Ilya
[jira] [Created] (IGNITE-9579) Document Random Forest
Aleksey Zinoviev created IGNITE-9579: Summary: Document Random Forest Key: IGNITE-9579 URL: https://issues.apache.org/jira/browse/IGNITE-9579 Project: Ignite Issue Type: Task Components: documentation, ml Reporter: Aleksey Zinoviev Assignee: Alexey Platonov Fix For: 2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9578) Document K-fold cross validation of models
Aleksey Zinoviev created IGNITE-9578: Summary: Document K-fold cross validation of models Key: IGNITE-9578 URL: https://issues.apache.org/jira/browse/IGNITE-9578 Project: Ignite Issue Type: Task Components: documentation, ml Reporter: Aleksey Zinoviev Assignee: Anton Dmitriev -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9577) Document Preprocessing
Aleksey Zinoviev created IGNITE-9577: Summary: Document Preprocessing Key: IGNITE-9577 URL: https://issues.apache.org/jira/browse/IGNITE-9577 Project: Ignite Issue Type: Task Components: documentation, ml Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev Fix For: 2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9576) Document Multi-Class Logistic Regression
Aleksey Zinoviev created IGNITE-9576: Summary: Document Multi-Class Logistic Regression Key: IGNITE-9576 URL: https://issues.apache.org/jira/browse/IGNITE-9576 Project: Ignite Issue Type: Task Components: documentation, ml Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev Fix For: 2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9575) Document Binary Logistic Regression
Aleksey Zinoviev created IGNITE-9575: Summary: Document Binary Logistic Regression Key: IGNITE-9575 URL: https://issues.apache.org/jira/browse/IGNITE-9575 Project: Ignite Issue Type: Task Components: documentation, ml Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev Fix For: 2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9574) Document Gradient boosting
Aleksey Zinoviev created IGNITE-9574: Summary: Document Gradient boosting Key: IGNITE-9574 URL: https://issues.apache.org/jira/browse/IGNITE-9574 Project: Ignite Issue Type: Task Components: documentation, ml Reporter: Aleksey Zinoviev Assignee: Alexey Platonov Fix For: 2.7 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] SomeFire opened a new pull request #8: Trigger build bugfix
SomeFire opened a new pull request #8: Trigger build bugfix URL: https://github.com/apache/ignite-teamcity-bot/pull/8 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ignite pull request #4746: Ignite 1.9.14
GitHub user mcherkasov opened a pull request: https://github.com/apache/ignite/pull/4746 Ignite 1.9.14 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-1.9.14 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4746.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4746 commit 37535634ef3325aaf9923fd17d24038dfd5cee38 Author: agura Date: 2017-07-11T13:24:54Z ignite-5722 Cache entries stay in onheap after scan query execution for OFFHEAP_TIRED cache with expiry policy commit c3e2eebeccbdc4bb3a7a0a70d09a8a7b63399c2c Author: Evgenii Zhuravlev Date: 2017-07-18T15:50:48Z IGNITE 5776 Add option to turn on filter reachable addresses in TcpCommunicationSpi commit 97d3f42c1c95a6aafce1d0c300ccfe6708398c17 Author: shtykh_roman Date: 2016-09-07T05:35:31Z IGNITE-3809: Fix for ArrayIndexOutOfBoundsException in GridUnsafeLru. (cherry picked from commit 31b9bb8) commit c2062d52a227dda5afee560d80c3bb4dd2ce09eb Author: dkarachentsev Date: 2017-07-19T05:41:46Z Remove empty test_utils.cpp commit 45cbba4853bab1ba4ffe2ea0d3add99a9d454aab Author: dkarachentsev Date: 2017-07-19T07:44:04Z IGNITE-5768 - Retry resolving class name from marshaller cache and .classname file. commit f24969f7e908645444df622642967a5f7fd3db23 Author: Evgenii Zhuravlev Date: 2017-07-19T16:30:07Z IGNITE 5775 JobsProcessor fix bug with delay in compute commit e5aab82f5629c2705e9bc82a7676f63c7c77062a Author: dkarachentsev Date: 2017-07-20T07:37:08Z Merge branch 'ignite-1.7.13' into ignite-1.8.9 # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/jdbc2/JdbcPreparedStatement.java # modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java commit 1544d3b63974ef8a96694cfc0681548fb4625a59 Author: Evgenii Zhuravlev Date: 2017-07-24T07:18:59Z Merge branch 'ignite-1.7.4-p2' into ignite-1.7.14 commit 7e5b479a9c48a6e272f8883f16d0ffa454e44046 Author: Andrey V. Mashenkov Date: 2017-07-24T10:27:52Z Merge remote-tracking branch 'origin/ignite-1.7.12-p1' into ignite-1.7.14 commit cc89a6a224ade08c92f44211dc144dc806c2b400 Author: dkarachentsev Date: 2017-07-26T08:19:19Z Merge branch 'ignite-1.8.9' into ignite-1.9.5 # Conflicts: # modules/core/src/main/java/org/apache/ignite/internal/MarshallerContextImpl.java # modules/core/src/test/java/org/apache/ignite/internal/processors/datastreamer/DataStreamerImplSelfTest.java # modules/indexing/src/test/java/org/apache/ignite/testsuites/IgniteCacheQuerySelfTestSuite2.java # modules/platforms/cpp/core-test/src/test_utils.cpp commit f3adb9559b42698771b0b9b5116dd535446d2bef Author: vsisko Date: 2017-07-26T08:40:19Z IGNITE-5781 Visor throws ClassCastException if cache store implementation is other than CacheJdbcPojoStore. commit a58688f6cc6c5b114dcdd1b2fde43b7e1e5e0732 Author: vsisko Date: 2017-07-26T08:40:19Z IGNITE-5781 Visor throws ClassCastException if cache store implementation is other than CacheJdbcPojoStore. (cherry picked from commit f3adb95) commit 5b8963e38254fcac3bdd255a95adf9f733ce08f2 Author: vsisko Date: 2017-07-26T08:40:19Z IGNITE-5781 Visor throws ClassCastException if cache store implementation is other than CacheJdbcPojoStore. (cherry picked from commit f3adb95) commit 6c6264e93c378ff3f083528b46e673fefc76f471 Author: vsisko Date: 2017-07-26T08:48:27Z Merge remote-tracking branch 'community/ignite-1.9.5' into ignite-1.9.5 commit b7d1fb25ceba20b82631bb2e926a0ad52bf19e9d Author: sboikov Date: 2017-07-20T14:43:17Z Do not process partition exchange messages in striped pool. (cherry picked from commit 3a33706) commit c338bb9f5ac8f34dccbac1f7058765c5ce4549a4 Author: sboikov Date: 2017-07-20T14:49:55Z Removed unnecessary discoCache.updateAlives. (cherry picked from commit 07a0698) commit e7fbe8fba6bcd427bebf6ab6fa8746bc47b0d42f Author: Andrey V. Mashenkov Date: 2017-07-31T14:20:29Z IGNITE-4800: Lucene query may fails with NPE. commit 8c992fb8ba33a0c0ac5c0fb741ee8ffd515c0f31 Author: Evgenii Zhuravlev Date: 2017-08-01T14:46:27Z IGNITE-5775 Fix removing jobs from activeJobs for jobAlwaysActivate commit 114f10a243e41ef1dec0f35bef92cbf6a9a4f161 Author: Evgenii Zhuravlev Date: 2017-08-01T14:47:25Z Merge branch 'ignite-1.7.14' of https://github.com/gridgain/apache-ignite into ignite-1.7.14 commit eb3471075035d4d6ed3f54b3c26e929a6a097262 Author: Igor Sapego Date: 2017-08-01T15:30:54Z Fix for C++ tests commit 9f12a2d4bf333840df50d2a56478349f530b65c8 Author: Andrey V. Mashenkov Date: 2017-08-02T11:23:26Z IGNITE-4800: Test fixed. commit
[GitHub] ignite pull request #4745: IGNITE-9411: mvcc timeouts
GitHub user pavlukhin opened a pull request: https://github.com/apache/ignite/pull/4745 IGNITE-9411: mvcc timeouts You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9411 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4745.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4745 commit db172dfc174614af7cbac8b5cecb2707b49e1bdc Author: ipavlukhin Date: 2018-09-04T10:53:41Z minors in CacheMvccSelectForUpdateQueryAbstractTest commit c288988dd4f3622c8539ab7f6f3f84b5c08b7a0c Author: ipavlukhin Date: 2018-09-04T11:13:01Z use timeout modes in deadlock tests commit e4bf4ae3bdcd8b049e66a90f30fb5720b34fd36c Author: ipavlukhin Date: 2018-09-12T10:10:50Z CacheMvccSqlLockTimeoutTest checking that waiting locks are timed out commit 3ad0289ade9964e8a9ec74e10f3f05ed96d72bfc Author: ipavlukhin Date: 2018-09-12T12:27:55Z translate to timeout exception in reducer commit 9d70e0ff36546ae05230481688b9a5f3b7bd7311 Author: ipavlukhin Date: 2018-09-12T12:29:53Z translate to timeout exception in DmlStatementsProcessor commit d34b1b88a498dd090efad058292f0ea19bbd5bef Author: ipavlukhin Date: 2018-09-12T13:31:58Z Merge branch 'master' into mvcc-lock-timeouts commit 0bb7981f9459a797cde4aa97fa200b3502b6b923 Author: ipavlukhin Date: 2018-09-12T14:28:06Z dirty conversion of remote timeout to TransactionTimeoutException, check new exception structure in test commit 9539ef913a8bdd364d8a449c3b4e8aecaace57ef Author: ipavlukhin Date: 2018-09-13T08:57:21Z establish first design for timeout exceptions propagation, put missing timeouts in place commit c7eea3664dec1aa09facee14ce24941dfa39b180 Author: ipavlukhin Date: 2018-09-13T09:59:34Z remove extra stack trace garbage from test commit 197f3e37c0cb2fa0337c880f2e66d55a878ce809 Author: ipavlukhin Date: 2018-09-13T10:31:22Z use common method for timeout calculation in DmlStatementsProcessor commit 980d43dcee8e8a540788c7305ad53280c88269c9 Author: ipavlukhin Date: 2018-09-13T12:06:58Z fix forgotten tx variable initialization in IgniteH2Indexing commit 75ba8821d844c8e7d250154b3fb68adaae842902 Author: ipavlukhin Date: 2018-09-13T12:15:51Z remove duplicated tests from suite ---
[GitHub] ignite pull request #4743: IGNITE-8149: fix for CI
Github user pavlukhin closed the pull request at: https://github.com/apache/ignite/pull/4743 ---
[GitHub] ignite pull request #4744: IGNITE-6454 reproducer fast finish
GitHub user dgarus opened a pull request: https://github.com/apache/ignite/pull/4744 IGNITE-6454 reproducer fast finish You can merge this pull request into a Git repository by running: $ git pull https://github.com/dgarus/ignite IGNITE-6454_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4744.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4744 commit 25b2834ec58283e42801acc0ec98a7e530cc9d86 Author: dgarus Date: 2018-09-13T12:07:47Z IGNITE-6454 reproducer fast finish ---
[jira] [Created] (IGNITE-9573) ZookeeperDiscoverySpiSaslFailedAuthTest fails after added to suite
Ilya Kasnacheev created IGNITE-9573: --- Summary: ZookeeperDiscoverySpiSaslFailedAuthTest fails after added to suite Key: IGNITE-9573 URL: https://issues.apache.org/jira/browse/IGNITE-9573 Project: Ignite Issue Type: Bug Reporter: Ilya Kasnacheev Assignee: Ilya Kasnacheev Due to proliferation of static fields in Zk and Security code it fails when a part of suite and/or will cause other tests to fail. Needs to be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4743: IGNITE-8149: fix for CI
GitHub user pavlukhin opened a pull request: https://github.com/apache/ignite/pull/4743 IGNITE-8149: fix for CI Fail DataStreamProcessorMvccSeflTest referring to IGNITE-9451 and mention same ticket in GridDhtPartitionsStateValidator. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-8149-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4743.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4743 commit 1fecf7bf7360f231778bb91a4d49b90693167119 Author: ipavlukhin Date: 2018-09-13T09:46:33Z fail DataStreamProcessorMvccSeflTest referring to IGNITE-9451 and mention same ticket in GridDhtPartitionsStateValidator ---
[jira] [Created] (IGNITE-9572) Web console: broken in Edge 17
Ilya Borisov created IGNITE-9572:
Summary: Web console: broken in Edge 17
Key: IGNITE-9572
URL: https://issues.apache.org/jira/browse/IGNITE-9572
Project: Ignite
Issue Type: Bug
Reporter: Ilya Borisov
Assignee: Ilya Borisov
What happens:
Edge 17 throws "{{Error: Expected ':'" and the web console does not work at
all.}}
{{What to do:}}
{{1. Investigate why this happens.}}
{{2. Fix the issue.}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Exchange stucks while node restoring state from WAL
Igniters, I need your help with the review of this patch. In general, it will help to reduce PME duration. I've moved binary memory recovery at the moment of node startup as we've discussed it with Pavel previously this topic. Changes are relatively small (+299 −95) and they are ready. I've left a comment in JIRA with high-level implementation details. Hope, this will help with the review [1]. Please, take a look. PR: https://github.com/apache/ignite/pull/4520/files Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-727 JIRA: https://issues.apache.org/jira/browse/IGNITE-7196 TC Run All: [2] [1] https://issues.apache.org/jira/browse/IGNITE-7196?focusedCommentId=16613175=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16613175 [2] https://ci.ignite.apache.org/viewLog.html?buildId=1835822=buildResultsDiv=IgniteTests24Java8_RunAll On Tue, 28 Aug 2018 at 17:59 Pavel Kovalenko wrote: > Hello Maxim, > > I think you're going in the right direction, but why you perform binary > recovery only in the case when a node in the baseline? > I think that node can also perform binary recovery before the first > exchange is started in case of > 1) First activation > 2) Baseline change > > In all of these cases, you need just local available CacheDescriptors to > restore binary state from WAL. > I would like also see a test when Ignite is stopped in the middle of a > checkpoint and binary recovery succeed before PME in that case also. > > > ср, 22 авг. 2018 г. в 15:31, Pavel Kovalenko : > > > Hello Maxim, > > > > Thank you for your work. I will review your changes within the next > > several days. > > > > 2018-08-22 11:12 GMT+03:00 Maxim Muzafarov : > > > >> Pavel, > >> > >> Thank you for so detailed introduction into the root of the problem of > >> ticket IGNITE-7196. > >> As you mentioned before, we can divide this ticket into two sub-tasks. > So, > >> I will > >> file new issue for `Logical consistency recovery`. I think it's more > >> convenient way > >> for reviewing and merging each of these high level Steps (PRs). > >> > >> Currently, let's focus on `Physical(binary) consistency recovery` as the > >> most critical > >> part of this issue and synchronize our vision and vision of other > >> community > >> members > >> on implementation details of Step 1. From this point everything what I'm > >> saying is about > >> only binary recovery. > >> > >> > >> I think PR is almost ready (I need to check TC carefully). > >> Can you look at my changes? Am I going the right way? > >> > >> PR: https://github.com/apache/ignite/pull/4520 > >> Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-727 > >> JIRA: https://issues.apache.org/jira/browse/IGNITE-7196 > >> > >> > >> These are my milestones which I've tried to solve: > >> 1. On first time cluster activation no changes required. We should keep > >> this behavior. > >> > >> 2. `startMemoryPolicies` earlier if need. > >> Now at the master branch data regions starts at onActivate method > called. > >> If we are trying > >> to restore binary state on node reconnects to cluster we should start > >> regions earlier. > >> > >> 3. `suspendLogging` method added to switch off handlers. > >> Keep fast PDS cleanup when joining is not in BLT (or belongs to another > >> BLT). > >> Restore memory and metastorage initialization required `WALWriter` and > >> `FileWriteHandle` > >> to be started. They can lock directory\files and block cleanups. > >> > >> 4. `isBaselineNode` check before resumming logging. > >> The same as point 3. Local node gets discovery data and determines > >> #isLocalNodeNotInBaseline. > >> If node not in current baseline it performs cleanup local PDS and we > >> should > >> reset state of > >> previously initialized at startup metastorage instance. > >> > >> 5. `cacheProcessorStarted` callback added to restore memory at startup. > >> We can restore memory only after having info about all > >> `CacheGroupDescriptor`. To achieve this > >> we should do restoring at the moment `GridCacheProcessor` started. > >> > >> 6. Move `fsync` for `MemoryRecoveryRecord` at node startup. > >> We are fsync'ing it inside running PME. Suppose, it used for recovery > >> actions and\or control > >> cluster state outside Ignite project (methods `nodeStart` and > >> `nodeStartedPointers` of this > >> are not used at Ignite project code). I've moved them at the node > startup, > >> but may be we should > >> dumping timestamp only at the time of node activation. Not sure. > >> > >> 7. `testBinaryRecoverBeforeExchange` test added to check restore before > >> PME. > >> Not sure that we should repeat wal reading logic as it was in > >> `testApplyDeltaRecords` test for > >> checking memory restoring. Suppose, minimally necessary condition is to > >> check `lastRestored` and > >> `checkpointHistory` states of database manager's (they should happend > >> before node joins to cluster). > >> > >> I will create separate ticket and refactor
[jira] [Created] (IGNITE-9571) Web console: update Project Structure UI to a new style
Pavel Konstantinov created IGNITE-9571: -- Summary: Web console: update Project Structure UI to a new style Key: IGNITE-9571 URL: https://issues.apache.org/jira/browse/IGNITE-9571 Project: Ignite Issue Type: Bug Reporter: Pavel Konstantinov Attachments: screenshot-1.png -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9570) Fix failures in CacheMvccSelectForUpdateQueryAbstractTest
Vladimir Ozerov created IGNITE-9570: --- Summary: Fix failures in CacheMvccSelectForUpdateQueryAbstractTest Key: IGNITE-9570 URL: https://issues.apache.org/jira/browse/IGNITE-9570 Project: Ignite Issue Type: Task Components: mvcc Reporter: Vladimir Ozerov Assignee: Vladimir Ozerov Fix For: 2.7 Caused by badly formed test itself. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9569) Web console: use $inject for DI instead of arrays
Ilya Borisov created IGNITE-9569:
Summary: Web console: use $inject for DI instead of arrays
Key: IGNITE-9569
URL: https://issues.apache.org/jira/browse/IGNITE-9569
Project: Ignite
Issue Type: Improvement
Components: wizards
Reporter: Ilya Borisov
Assignee: Ilya Borisov
To do:
1. Remove provider registration by array spread, like this:
{code:java}
.service(...serviceArray){code}
Instead, use the canonical AngularJS approach:
{code:java}
.service('SeriveName', Service){code}
2. Do not use array Di syntax for exported symbols:
{code:java}
export ['IgniteVersion', 'Confirm', function directive (version,
confirm){}]{code}
Instead, use $inject property:
{code:java}
export function directive(version, confirm) {}
directive.$inject = ['IgniteVersion', 'Confirm']{code}
Motivation:
The changes above will make older providers accessible to TypeScript, which in
turn will allow to increase type coverage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (IGNITE-9568) Web console: fields are not visible under Firefox
Pavel Konstantinov created IGNITE-9568: -- Summary: Web console: fields are not visible under Firefox Key: IGNITE-9568 URL: https://issues.apache.org/jira/browse/IGNITE-9568 Project: Ignite Issue Type: Bug Reporter: Pavel Konstantinov Attachments: screenshot-1.png -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Class field ThreadLocal. Why not static?
Maxim, If multiple instances of Ignite is started in the same JVM and a user thread will access first one instance of Ignite, then another, you will end up with the static thread local holding the last WAL pointer from the second grid. This is possible, for example, when a user thread commits a transaction or runs an atomic update on a data node. Any access of the first Ignite instance will have an invalid thread-local value. вт, 11 сент. 2018 г. в 13:29, Maxim Muzafarov : > Alexey, Ivan, > > Agree. Keeping strong references to the Thread object is the source of > memory leak with ThreadLocals variables > and the values that it stores. ThreadLocalMap is bound to the Thread > lifespan [1], so I think when we are using > everything right all will be GC'ed correctly. > Is this memory leaks with ThreadLocal's you mean, Alexey? If not, please, > share your example. > > Also, agree that these usages should be bound to the component lifespan. > But for `FileWriteAheadLogManager` > I think this variable used not semantically right. I've dumped all threads > (total ~49 threads) > that are using `lastWalPtr` in `FileWriteAheadLogManager`. For instance, > * exchange-worker-#40%wal.IgniteWalRecoveryTest0% > * sys-#148%wal.IgniteWalRecoveryTest1% > * db-checkpoint-thread-#129%wal.IgniteWalRecoveryTest2% > Suppose everything would be OK here for `static` and `non-static` case of > ThreadLocal. > > [1] > > http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/java/lang/Thread.java#l760 > > On Tue, 11 Sep 2018 at 13:05 Павлухин Иван wrote: > > > Dmitriy, > > > > Could you point to some piece of code implementing described pattern? > > > > 2018-09-11 13:02 GMT+03:00 Павлухин Иван : > > > > > Alex, > > > > > > ThreadLocal subclass is used in IgniteH2Indexing for simple access to > H2 > > > Connection from current thread. Such subclass has a capability to > create > > > connection if one does not exist, so obtaining connection is merely > > > ThreadLocal.get. Also there are scheduled routines to cleanup > connections > > > and associated with them statement cache after some expiration time. > For > > > that reason Map is maintained. As query > can > > > run on user thread we need to cleanup mentioned map to avoid a leak > when > > > Thread is terminated. So we need to check thread status in cleanup > > routines > > > and remove entries for terminated Threads. And historically there was > no > > > cleanup for terminated threads and leak was possible. And also great > care > > > must be taken in order to avoid cyclic reference between ThreadLocal > > > instance and a stored value. Which easily could occur if the stored > value > > > is covered by multiple layers of abstraction. > > > > > > And I am describing some historical state. Now machinery in > > IgniteH2Indexing > > > is even more complex (I hope we will have a chance to improve it). > > > > > > 2018-09-11 11:00 GMT+03:00 Alexey Goncharuk < > alexey.goncha...@gmail.com > > >: > > > > > >> Ivan, > > >> > > >> Can you elaborate on the issue with the thread local cleanup you've > > faced? > > >> > > >> вт, 11 сент. 2018 г. в 9:13, Павлухин Иван : > > >> > > >> > Guys, > > >> > > > >> > As we know ThreadLocal is an instrument which should be used with > > great > > >> > care. And I recently faced with problems related to proper cleanup > of > > >> > ThreadLocal which is not needed anymore. In my opinion the best > thing > > >> (in > > >> > ideal world) is to get rid of ThreadLocal where possible, but I > guess > > >> that > > >> > it is quite hard (in real world). > > >> > > > >> > Also, a question comes to my mind. As ThreadLocal is so common in > our > > >> code, > > >> > could you suggest some guidance or code fragments which address > proper > > >> > ThreadLocal > > >> > lifecycle control and especially cleanup? > > >> > > > >> > 2018-09-10 12:46 GMT+03:00 Alexey Goncharuk < > > alexey.goncha...@gmail.com > > >> >: > > >> > > > >> > > Maxim, > > >> > > > > >> > > Ignite supports starting multiple instances of Ignite in the same > > VM, > > >> so > > >> > > having static thread locals for the fields you mentioned does not > > >> work. > > >> > > > > >> > > Generally, I think thread-local should be bound to the lifespan of > > the > > >> > > component it describes. Static thread-locals are hard to clean-up > > and > > >> > they > > >> > > often lead to leaks, so I would rather changed existing static > > >> > > thread-locals to be non-static. > > >> > > > > >> > > --AG > > >> > > > > >> > > пн, 10 сент. 2018 г. в 11:54, Maxim Muzafarov >: > > >> > > > > >> > > > Igniters, > > >> > > > > > >> > > > According to javadoc [1] class ThreadLocal: > > >> > > > `ThreadLocal instances are typically private *static* fields in > > >> classes > > >> > > > that wish to associate state with a thread (e.g., a user ID or > > >> > > Transaction > > >> > > > ID).` > > >> > > > > > >> > > > So, AFAIK non-static ThreadLocal usage means as `per thread - > per > > >>
