Re: Apache Ignite 2.7. Last Mile
Hello, Igniters. Changes regarding to GridToStringBuilder reverted in ignite-2.7 branch: * IGNITE-8493 * IGNITE-9209 * IGNITE-602 We have 1 ticket for 2.7: IGNITE-10393: DataStreamer failed with NPE for MVCC caches which is unassigned. Who can fix it? В Вт, 20/11/2018 в 12:38 +0300, Dmitrii Ryabov пишет: > Yes, revert both. > > вт, 20 нояб. 2018 г., 11:52 Vladimir Ozerov voze...@gridgain.com: > > > +1 for reverting both. > > > > On Tue, Nov 20, 2018 at 9:43 AM Nikolay Izhikov > > wrote: > > > > > Hello, Dmitrii. > > > > > > I see 2 tickets for this improvement: > > > > > > IGNITE-602 - [Test] GridToStringBuilder is vulnerable for > > > StackOverflowError caused by infinite recursion [1] > > > IGNITE-9209 - GridDistributedTxMapping.toString() returns broken string > > > > [2] > > > > > > Should we revert both commits? > > > > > > [1] https://github.com/apache/ignite/commit/d67c5bf > > > [2] https://github.com/apache/ignite/commit/9bb9c04 > > > > > > > > > В Пн, 19/11/2018 в 13:36 +0300, Dmitrii Ryabov пишет: > > > > I agree to revert and make fix for 2.8. So, we will have more time to > > > > > > test > > > > it. > > > > > > > > пн, 19 нояб. 2018 г., 10:53 Vladimir Ozerov voze...@gridgain.com: > > > > > > > > > +1 for revert. > > > > > > > > > > On Sun, Nov 18, 2018 at 11:31 PM Dmitriy Pavlov > > > > > wrote: > > > > > > > > > > > I personally don't mind. > > > > > > > > > > > > But I would like Dmitry Ryabov and Alexey Goncharuck share their > > > > > > > > > > opinions. > > > > > > > > > > > > вс, 18 нояб. 2018 г., 20:43 Nikolay Izhikov : > > > > > > > > > > > > > Yes, I think so. > > > > > > > > > > > > > > вс, 18 нояб. 2018 г., 20:34 Denis Magda dma...@apache.org: > > > > > > > > > > > > > > > Sounds good to me. Are we starting the vote then? > > > > > > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Nov 18, 2018 at 8:25 AM Nikolay Izhikov < > > > > > > nizhi...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello, Igniters. > > > > > > > > > > > > > > > > > > This issue is the only ticket that blocks 2.7 release. > > > > > > > > > > > > > > > > > > I looked at IGNITE-602 PR and GridToStringBuilder. > > > > > > > > > The code looks complicated for me. > > > > > > > > > And it's not obvious for me how to fix this issue in a short > > > > > > period > > > > > > > > > > > > of > > > > > > > > > time. > > > > > > > > > Especially, code deals with recursion and other things that > > > > can > > > > > > > > > > lead > > > > > > to > > > > > > > > > very dangerous errors. > > > > > > > > > > > > > > > > > > Let's revert this patch and fix it in calmly. > > > > > > > > > Also, we need additional tests for it. > > > > > > > > > > > > > > > > > > В Пт, 16/11/2018 в 17:57 +0300, Dmitrii Ryabov пишет: > > > > > > > > > > Ok, I'll check the issue. > > > > > > > > > > пт, 16 нояб. 2018 г. в 17:52, Alexey Goncharuk < > > > > > > > > > > > > > > > > > > alexey.goncha...@gmail.com>: > > > > > > > > > > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > > > > > > > > > I've just found that S.toString() implementation is > > > > broken > > > in > > > > > > > > > > > > > > > > > > ignite-2.7 and master [1]. It leads to a message > > > > > > > > > > > Wrapper [p=Parent [a=0]Child [b=0, super=]] > > > > > > > > > > > being formed instead of > > > > > > > > > > > Wrapper [p=Child [b=0, super=Parent [a=0]]] > > > > > > > > > > > for classes with inheritance that use > > > > > > > > > > S.toString(SomeClass.class, > > > > > > > > > this, super.toString()) embedded to some other object. > > > > > > > > > > > > > > > > > > > > > > Dmitrii Ryabov, I've reverted two commits related to > > > > > > IGNITE-602 > > > > > > > > > > > > and > > > > > > > > > IGNITE-9209 tickets locally and it fixes the issue. Can you > > > > > > take a > > > > > > > > > > > > look > > > > > > > > at > > > > > > > > > the issue? > > > > > > > > > > > > > > > > > > > > > > I think this regression essentially makes our logs > > > > > > unreadable > > > > > > > > > > in > > > > > > > some > > > > > > > > > cases and I would like to get it fixed in ignite-2.7 or > > > > revert > > > both > > > > > > > > > > > > > > > > commits > > > > > > > > > from the release. > > > > > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10301 > > > > > > > > > > > > > > > > > > > > > > пт, 9 нояб. 2018 г. в 09:22, Nikolay Izhikov < > > > > > > > > > > > > nizhi...@apache.org > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > Hello, Igniters. > > > > > > > > > > > > > > > > > > > > > > > > We still have 5 tickets for 2.7: > > > > > > > > > > > > > > > > > > > > > > > > IGNITE-10052Andrew Mashenkov Restart node during TX > > > > > > > > > > causes > > > > > > > > > vacuum error. > > > > > > > > > > > > IGNITE-10170Unassigned .NET: > > > > > > > > > > > >
[GitHub] ignite pull request #5492: ignite-2.7 revert GridToStringBuilder changes
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5492 ---
[MTCGA]: new failures in builds [2389319] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *New test failure in master CacheContinuousQueryAsyncFailoverMvccTxSelfTest.testFailoverStartStopBackup https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=318679578210209893=%3Cdefault%3E=testDetails Changes may lead to failure were done by - alexey.goncharuk https://ci.ignite.apache.org/viewModification.html?modId=840333 - gvvinblade https://ci.ignite.apache.org/viewModification.html?modId=840312 - oignatenko https://ci.ignite.apache.org/viewModification.html?modId=840298 - vpyatkov https://ci.ignite.apache.org/viewModification.html?modId=840250 - ivandasch https://ci.ignite.apache.org/viewModification.html?modId=840342 - alexey.goncharuk https://ci.ignite.apache.org/viewModification.html?modId=840227 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 07:38:22 24-11-2018
[jira] [Created] (IGNITE-10398) CacheMetrics always return 0 for local cache
Evgenii Zhuravlev created IGNITE-10398:
--
Summary: CacheMetrics always return 0 for local cache
Key: IGNITE-10398
URL: https://issues.apache.org/jira/browse/IGNITE-10398
Project: Ignite
Issue Type: Bug
Reporter: Evgenii Zhuravlev
However, it shows the right offHeapEntriesCnt.
Short code snippet:
{code:java}
IgniteConfiguration igniteConfig = new IgniteConfiguration();
CacheConfiguration cacheConfig = new CacheConfiguration("testCache");
cacheConfig.setStatisticsEnabled(true);
igniteConfig.setCacheConfiguration(cacheConfig);
cacheConfig.setCacheMode(CacheMode.LOCAL);
try (Ignite ignite = Ignition.start(igniteConfig)) {
IgniteCache cache = ignite.getOrCreateCache(cacheConfig.getName());
cache.put("key", "val");
cache.put("key2", "val2");
cache.remove("key2");
System.out.println(cache.localMetrics());
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] dspavlov opened a new pull request #82: Ignite 9542 remove unused code
dspavlov opened a new pull request #82: Ignite 9542 remove unused code URL: https://github.com/apache/ignite-teamcity-bot/pull/82 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: Thin clients all in one
Stepan,
Sorry, I forgot to update from upstream prior to start working on this
issue, and thus brought a regression. My bad. Just merged with the
latest master. Please, check it out again.
Dmitry
On 11/24/18 1:37 AM, Stepan Pilschikov wrote:
Dmitry,
Iv checked and its actually work
But a specially in this branch i found another bug
Please look at my last comment:
https://issues.apache.org/jira/browse/IGNITE-10358?focusedCommentId=16697285=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16697285
пт, 23 нояб. 2018 г. в 01:21, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:
Stepan,
Thank you for your great job in evaluating Python thin client, as well
as other thin clients.
There was indeed a bug in Python client regarding the handling of type
hints in Collection type. I created a fix and did a PR under
IGNITE-10358 task, but the same PR is also fixes the problem in
IGNITE-10230 task.
As of handling the type mapping in gists you provided, I left comments
on both tasks.
Dmitry
On 11/21/18 6:37 PM, Stepan Pilschikov wrote:
Dmitry, Alexey
Thank you for help, this answers help me a lot with understanding how
clients are work
Not so long time ago i met problem which is have expected behavior, but
its
may broke some workflows in future for some users
Its all about not specified data types in collections and map's
All description and examples in
https://issues.apache.org/jira/browse/IGNITE-10358
Dmitry, can you have a quick look at it and maybe in future somehow fix
it?
пт, 26 окт. 2018 г. в 19:05, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:
Stepan!
TL/DR: what you got with Python client in your gist is an intended
behavior.
Explanation: As per docs, Object array contains of type ID (which is
defaults to -1) and an array of objects.
https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-object-array
Your gist might be fixed accordingly:
```
from pyignite import Client
from pyignite.datatypes import *
OBJECT_ARRAY_TYPE_ID = -1
OBJECT_ARRAY_CONTENTS = [1, 2]
client = Client()
client.connect('127.0.0.1', 10800)
cache = client.get_or_create_cache("PY_OBJECT_ARRAY")
cache.put(
1,
(OBJECT_ARRAY_TYPE_ID, OBJECT_ARRAY_CONTENTS),
key_hint=IntObject,
value_hint=ObjectArrayObject,
)
# Python output: print(cache.get(1))
# (-1, [1, 2])
```
The situation is similar with Map and Collection, they have types. Types
and type IDs are mostly useless in Python, but I left them for
interoperability reasons. If you think I should kick them out, just let
me know.
The usage of these 3 data types is documented and tested. You can refer
to the table in “Data types” section:
https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/datatypes/parsers.html
The tests are here:
https://github.com/apache/ignite/blob/master/modules/platforms/python/tests/test_datatypes.py#L116-L124
On 10/26/18 11:57 PM, Stepan Pilschikov wrote:
Hi, everyone
Create new thread to centralize cross compatibility and others common
problems between thin clients
Tying to use Object array to exchange different data between JS, PHP
and
Python thin clients
JS and PHP simply can't put any type of arrays
Python can put data, but if you take it, data would be completely
different, maybe during this put process data is changed
Code and output:
PHP -
https://gist.github.com/pilshchikov/e887d31d4f2f36923470fead14c7801a
JS -
https://gist.github.com/pilshchikov/ba49067fd8924ebdf4414ec63838b86d
Python -
https://gist.github.com/pilshchikov/f4bbf76e31547e2dca7d4cc5d55bd24f
I'm tried different data types (string, double, float, complex objects,
just random objects, char, byte, Date), still
How, from my perspective, it should works:
put array of any type and then get this array.
Example: put [1,2,3] -> get [1,2,3] or put [new Date(), new Date()] ->
get [[Date object], [Date object]] (like in java thin client)
Looks like bug in all clients, what you think?
I wanted to try Collections, but i think this type have same problem
[GitHub] ignite pull request #5471: IGNITE-10216: disable sort annotations in inspect...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5471 ---
[GitHub] ignite pull request #5492: ignite-2.7 revert GridToStringBuilder changes
GitHub user nizhikov opened a pull request: https://github.com/apache/ignite/pull/5492 ignite-2.7 revert GridToStringBuilder changes Revert of - IGNITE-8493 - IGNITE-9209 - IGNITE-602 You can merge this pull request into a Git repository by running: $ git pull https://github.com/nizhikov/ignite ignite-2.7-revert-toString Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5492.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5492 commit e6b42abb81c22bd119a9233023be6be66b2aad09 Author: Nikolay Izhikov Date: 2018-11-23T19:51:08Z Revert "IGNITE-8493 GridToStringBuilder arrayToString refactoring. - Fixes #4501." This reverts commit dd8c933fd44e4ad9b315daccce8e2327606867b0. commit 5f968d2413c0c914da0ebcb5c051023429b3e518 Author: Nikolay Izhikov Date: 2018-11-23T19:51:45Z Revert "IGNITE-9209 fix for GridDistributedTxMapping.toString() returns broken string. - Fixes #4519." This reverts commit 9bb9c043513c3cc6c6b70c6c3395e5bb76fad75e. commit 307ac58d1aa1f0145c16affe6de96314b1c8ecef Author: Nikolay Izhikov Date: 2018-11-23T19:52:12Z Revert "IGNITE-602 Fixed possible StackOverflowError in GridToStringBuilder when circular references are present - Fixes #1558." This reverts commit d67c5bf4c22338695a116e0fbf0a58a4d581af5d. ---
[GitHub] ignite pull request #5455: Ignite 2.7 with tde fixes
Github user nizhikov closed the pull request at: https://github.com/apache/ignite/pull/5455 ---
[GitHub] ignite pull request #5488: IGNITE-10392 : Use lastAffChangedTopVer if DiscoC...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5488 ---
[GitHub] ignite pull request #5426: IGNITE-9996: Final fix
Github user nizhikov closed the pull request at: https://github.com/apache/ignite/pull/5426 ---
[GitHub] ignite pull request #5491: IGNITE-10397 Fix for 2.5
GitHub user Jokser opened a pull request: https://github.com/apache/ignite/pull/5491 IGNITE-10397 Fix for 2.5 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10397-2.5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5491.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5491 commit 702c31859251738b187e8db2a8155264f39b8b52 Author: Ivan Daschinskiy Date: 2018-07-23T12:29:08Z IGNITE-8820 Fix rollback logic when tx is initiated from client. - Fixes #4399. Signed-off-by: Alexey Goncharuk (cherry picked from commit 5794eb0) commit 32777e77797f4cf3260d42d4ca408336247d4e55 Author: Dmitriy Govorukhin Date: 2018-07-23T15:01:37Z IGNITE-9049 Fixed write of SWITCH_SEGMENT_RECORD at the end of a segment file - Fixes #4401. Signed-off-by: Alexey Goncharuk (cherry picked from commit 713a428) commit bbf8f83afe3b07e807912d9bab78873edf698d4d Author: Andrey V. Mashenkov Date: 2018-07-24T14:38:34Z IGNITE-8892 Fixed CacheQueryFuture usage in DataStructures processor - Fixes #4415. Signed-off-by: Alexey Goncharuk (cherry picked from commit 281a400) commit e0404913088b5cd97cd79de1b62e6aaa4c00b5d2 Author: Andrey V. Mashenkov Date: 2018-07-24T15:24:57Z Merge remote-tracking branch 'origin/ignite-2.5.1-master' into ignite-2.5.1-master commit 86c52700a007368a351ac3595a8e38bb02923080 Author: Sergey Chugunov Date: 2018-07-25T13:26:12Z IGNITE-9040 StopNodeFailureHandler is not able to stop node correctly on node segmentation Signed-off-by: Andrey Gura (cherry-picked from commit#469aaba59c0539507972f4725642b2f2f81c08a0) commit e1c3cc9d30baeaff4e653751775ab39a347b753b Author: Pavel Kovalenko Date: 2018-07-24T14:48:57Z IGNITE-8791 Fixed missed update counter in WAL data record for backup transaction - Fixes #4264. Signed-off-by: Alexey Goncharuk (cherry picked from commit 13e2a31) commit 2ea5b376d895fe41cc882c4383b5d3d2793ba684 Author: devozerov Date: 2018-07-31T09:53:51Z IGNITE-9114: SQL: fail query after some timeout if it cannot be mapped to topology. This closes #4453. commit 5b230b7015148e483136c229c9e6081bb9c68023 Author: Denis Mekhanikov Date: 2018-04-20T15:41:06Z IGNITE-8134 Subscribe to system cache events on nodes outside BLT Signed-off-by: Andrey Gura (cherry picked from commit c82277e) commit 2528435ec946c479f6bf4eb7eaeabf092a0536a3 Author: devozerov Date: 2018-07-31T14:15:49Z IGNITE-9114: SQL: use query time in retry timeout calculation. This closes #4460. commit 98d0ccc729e647a4ad8876e2d29423904e6fa6ca Author: devozerov Date: 2018-07-31T14:18:49Z Merge remote-tracking branch 'upstream/ignite-2.5.1-master' into ignite-2.5.1-master commit c78a7f215699f65983edfcc0c9014d9e3108f381 Author: Anton Kalashnikov Date: 2018-08-01T09:23:03Z IGNITE-8757 idle_verify utility doesn't show both update counter and hash conflicts (cherry picked from commit 9131e4d) commit 66369583096b04d71c3f82e1ebfd1922ae6b0b6a Author: Anton Kalashnikov Date: 2018-07-31T13:13:27Z IGNITE-8973 Calculate partition hash and print into standard output Signed-off-by: Andrey Gura (cherry picked from commit 8309cef) commit e92788d392d4c60f76c4fd0f38c52669f4e3e772 Author: Evgeny Stanilovskiy Date: 2018-08-01T11:05:58Z IGNITE-8939 Catch and proper propagate transaction string reprsentation - Fixes #4454. Signed-off-by: Dmitriy Pavlov (cherry picked from commit 458480c5b0520aa8e28935361a37ab49e1e65ff6) commit 60155cb55ca35790da7f80ae167dd7fb3f3793fd Author: Andrey V. Mashenkov Date: 2018-08-01T10:23:32Z IGNITE-9154: Fixed conflict version passed to events for ATOMIC cache. This closes #4463. (cherry picked from commit 3ea3a56) commit 15d72fc022f64939fee336438256460024ca59c0 Author: Andrey V. Mashenkov Date: 2018-08-01T12:58:22Z Merge remote-tracking branch 'origin/ignite-2.5.1-master' into ignite-2.5.1-master commit 158f36de551e860176ac7f3bcf92e9751039a35c Author: ilantukh Date: 2018-08-01T13:31:15Z IGNITE-8900 SqlFieldsQuery provides incorrect result when item size exceeds page size. (cherry picked from commit f1ecbbc) commit ef331f3ba58da49746f34546f1972cf65c083b3d Author: Maxim Muzafarov Date: 2018-08-01T15:39:54Z IGNITE-7165 Re-balancing is cancelled if client node joins Signed-off-by: Anton Vinogradov (cherry picked from commit 137dd06aaee9cc84104e6b4bf48306b050341e3a) commit cfe95a1b1762501dd9ac48a1a24228ac0bc3b5c3 Author: Pavel Kovalenko Date: 2018-08-02T12:51:17Z IGNITE-9111 Do not wait for deactivation. Signed-off-by: agura
[jira] [Created] (IGNITE-10397) SQL Schema may be lost after cluster activation and simple query run
Pavel Kovalenko created IGNITE-10397:
Summary: SQL Schema may be lost after cluster activation and
simple query run
Key: IGNITE-10397
URL: https://issues.apache.org/jira/browse/IGNITE-10397
Project: Ignite
Issue Type: Bug
Affects Versions: 2.8
Reporter: Pavel Kovalenko
Assignee: Pavel Kovalenko
Fix For: 2.8
Scenario:
1) Start 3 grids in a multithread mode with auto-activation.
2) Start the client.
3) Run a simple query like this
{noformat}
cache(DEFAULT_CACHE_NAME + 0).query(new SqlQuery<>(Integer.class,
"1=1")).getAll();
{noformat}
Exception with the message that schema not found will be thrown:
{noformat}
[2018-11-23
19:56:57,284][ERROR][query-#223%distributed.CacheMessageStatsIndexingTest2%][GridMapQueryExecutor]
Failed to execute local query.
class org.apache.ignite.internal.processors.query.IgniteSQLException: Failed to
set schema for DB connection for thread [schema=default0]
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.connectionForThread(IgniteH2Indexing.java:549)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.connectionForSchema(IgniteH2Indexing.java:392)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:767)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:637)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onMessage(GridMapQueryExecutor.java:224)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor$2.onMessage(GridMapQueryExecutor.java:184)
at
org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:2333)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.h2.jdbc.JdbcSQLException: Schema "default0" not found; SQL
statement:
SET SCHEMA "default0" [90079-195]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.engine.Database.getSchema(Database.java:1755)
at org.h2.command.dml.Set.update(Set.java:408)
at org.h2.command.CommandContainer.update(CommandContainer.java:101)
at org.h2.command.Command.executeUpdate(Command.java:260)
at org.h2.jdbc.JdbcStatement.executeUpdateInternal(JdbcStatement.java:137)
at org.h2.jdbc.JdbcStatement.executeUpdate(JdbcStatement.java:122)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.connectionForThread(IgniteH2Indexing.java:541)
... 13 more
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ignite pull request #5475: IGNITE-10339 Skip index partition file integrity ...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5475 ---
[jira] [Created] (IGNITE-10396) IgniteCache Key class with HashMap field doesn't recognized in some cases
PetrovMikhail created IGNITE-10396: -- Summary: IgniteCache Key class with HashMap field doesn't recognized in some cases Key: IGNITE-10396 URL: https://issues.apache.org/jira/browse/IGNITE-10396 Project: Ignite Issue Type: Bug Environment: [^IgniteCacheRemoveTest.java] Reporter: PetrovMikhail Attachments: IgniteCacheRemoveTest.java IgniteCache#containsKey or IgniteCache#remove methods return false when in their parameters passed instance of key class produced via cache iterator. Even in so situation when IgniteCache.Entry with such key is existing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #5483: IGNITE-10381 Fixed U.doInParallel early terminati...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5483 ---
Re: Thin clients all in one
Dmitry,
Iv checked and its actually work
But a specially in this branch i found another bug
Please look at my last comment:
https://issues.apache.org/jira/browse/IGNITE-10358?focusedCommentId=16697285=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16697285
пт, 23 нояб. 2018 г. в 01:21, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com>:
> Stepan,
>
> Thank you for your great job in evaluating Python thin client, as well
> as other thin clients.
>
> There was indeed a bug in Python client regarding the handling of type
> hints in Collection type. I created a fix and did a PR under
> IGNITE-10358 task, but the same PR is also fixes the problem in
> IGNITE-10230 task.
>
> As of handling the type mapping in gists you provided, I left comments
> on both tasks.
>
> Dmitry
>
> On 11/21/18 6:37 PM, Stepan Pilschikov wrote:
> > Dmitry, Alexey
> >
> > Thank you for help, this answers help me a lot with understanding how
> > clients are work
> >
> > Not so long time ago i met problem which is have expected behavior, but
> its
> > may broke some workflows in future for some users
> >
> > Its all about not specified data types in collections and map's
> > All description and examples in
> > https://issues.apache.org/jira/browse/IGNITE-10358
> >
> > Dmitry, can you have a quick look at it and maybe in future somehow fix
> it?
> >
> > пт, 26 окт. 2018 г. в 19:05, Dmitry Melnichuk <
> > dmitry.melnic...@nobitlost.com>:
> >
> >> Stepan!
> >>
> >> TL/DR: what you got with Python client in your gist is an intended
> >> behavior.
> >>
> >> Explanation: As per docs, Object array contains of type ID (which is
> >> defaults to -1) and an array of objects.
> >>
> >>
> >>
> https://apacheignite.readme.io/docs/binary-client-protocol-data-format#section-object-array
> >>
> >> Your gist might be fixed accordingly:
> >>
> >> ```
> >> from pyignite import Client
> >> from pyignite.datatypes import *
> >>
> >> OBJECT_ARRAY_TYPE_ID = -1
> >> OBJECT_ARRAY_CONTENTS = [1, 2]
> >>
> >> client = Client()
> >> client.connect('127.0.0.1', 10800)
> >> cache = client.get_or_create_cache("PY_OBJECT_ARRAY")
> >> cache.put(
> >> 1,
> >> (OBJECT_ARRAY_TYPE_ID, OBJECT_ARRAY_CONTENTS),
> >> key_hint=IntObject,
> >> value_hint=ObjectArrayObject,
> >> )
> >>
> >> # Python output: print(cache.get(1))
> >> # (-1, [1, 2])
> >> ```
> >>
> >> The situation is similar with Map and Collection, they have types. Types
> >> and type IDs are mostly useless in Python, but I left them for
> >> interoperability reasons. If you think I should kick them out, just let
> >> me know.
> >>
> >> The usage of these 3 data types is documented and tested. You can refer
> >> to the table in “Data types” section:
> >>
> >>
> >>
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/datatypes/parsers.html
> >>
> >> The tests are here:
> >>
> >>
> >>
> https://github.com/apache/ignite/blob/master/modules/platforms/python/tests/test_datatypes.py#L116-L124
> >>
> >> On 10/26/18 11:57 PM, Stepan Pilschikov wrote:
> >>> Hi, everyone
> >>>
> >>> Create new thread to centralize cross compatibility and others common
> >>> problems between thin clients
> >>>
> >>> Tying to use Object array to exchange different data between JS, PHP
> and
> >>> Python thin clients
> >>>
> >>> JS and PHP simply can't put any type of arrays
> >>> Python can put data, but if you take it, data would be completely
> >>> different, maybe during this put process data is changed
> >>>
> >>> Code and output:
> >>> PHP -
> >> https://gist.github.com/pilshchikov/e887d31d4f2f36923470fead14c7801a
> >>> JS -
> >> https://gist.github.com/pilshchikov/ba49067fd8924ebdf4414ec63838b86d
> >>> Python -
> >>> https://gist.github.com/pilshchikov/f4bbf76e31547e2dca7d4cc5d55bd24f
> >>>
> >>> I'm tried different data types (string, double, float, complex objects,
> >>> just random objects, char, byte, Date), still
> >>>
> >>> How, from my perspective, it should works:
> >>> put array of any type and then get this array.
> >>> Example: put [1,2,3] -> get [1,2,3] or put [new Date(), new Date()] ->
> >>> get [[Date object], [Date object]] (like in java thin client)
> >>>
> >>> Looks like bug in all clients, what you think?
> >>>
> >>> I wanted to try Collections, but i think this type have same problem
> >>
> >
>
>
[GitHub] ignite pull request #5490: IGNITE-8718: Updated C++ doxygen comments about B...
GitHub user isapego opened a pull request: https://github.com/apache/ignite/pull/5490 IGNITE-8718: Updated C++ doxygen comments about BinaryWriter/BinaryReader You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-8718 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5490.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5490 commit a8d8637bb342a6e4e254d755fbcd11d90a02ba97 Author: Igor Sapego Date: 2018-11-23T14:31:13Z IGNITE-8718: Fixed comments for BinaryWriter commit 79729136d648bd7c65a68cf67e9017c20bb32b7f Author: Igor Sapego Date: 2018-11-23T15:02:26Z IGNITE-8718: Added comments for BinaryReader and BinaryRawReader ---
[GitHub] ignite pull request #4785: IGNITE-9550 Get operation returns null for a lost...
Github user ibessonov closed the pull request at: https://github.com/apache/ignite/pull/4785 ---
[jira] [Created] (IGNITE-10395) Add to control.sh --cache --tx overall info:
ARomantsov created IGNITE-10395: --- Summary: Add to control.sh --cache --tx overall info: Key: IGNITE-10395 URL: https://issues.apache.org/jira/browse/IGNITE-10395 Project: Ignite Issue Type: Bug Reporter: ARomantsov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: How to deprecate unused Ignite's system property properly? (IGNITE-7441 Drop IGNITE_SERVICES_COMPATIBILITY_MODE system property)
Denis, thank you for the reply. I prepared PR [1] for the task [2]. Tests look good to me (the same failures in comparison with master branch). Could you review the PR please (it should not take a lot of time)? [1] https://github.com/apache/ignite/pull/5482 [2] https://issues.apache.org/jira/browse/IGNITE-7441 On Thu, Nov 22, 2018 at 5:58 PM Denis Mekhanikov wrote: > > Vyacheslav, > > You are right. This property is not used anywhere, so you can safely remove > it. > I don't think, there is any need in deprecation. You can just go ahead and > drop it, > since it doesn't have any effect. > > Denis > > чт, 22 нояб. 2018 г. в 15:47, Vyacheslav Daradur : > > > Hi, Igniters! > > > > Here is Jira issue [1] to drop one of Ignite's system property > > "IGNITE_SERVICES_COMPATIBILITY_MODE" because it is not used. > > > > I looked through git history and related Jira issues, the common > > conclusions: > > - the property was introduced in Ignite 1.7 within the task [2] to use > > LazyServiceConfiguration with premarshaled services instance; > > - Ignite 2.* was released which is incompatible with 1.*; > > - since Ignite 2.3 the property is completely ignored after introduced > > batch deployment mode [3]; > > > > Looks like we can just remove the property without introducing any > > compatibility issues. > > Also, related node attribute 'ATTR_SERVICES_COMPATIBILITY_MODE' can be > > safely removed. > > > > So, my question is: can I just remove following properties or I should > > deprecate them and remove all usages? > > IgniteSystemProperties#IGNITE_SERVICES_COMPATIBILITY_MODE > > IgniteNodeAttributes#ATTR_SERVICES_COMPATIBILITY_MODE > > > > [1] https://issues.apache.org/jira/browse/IGNITE-7441 > > [2] https://issues.apache.org/jira/browse/IGNITE-3056 > > [3] https://issues.apache.org/jira/browse/IGNITE-5145 > > > > -- > > Best Regards, Vyacheslav D. > > -- Best Regards, Vyacheslav D.
[jira] [Created] (IGNITE-10393) DataStreamer failed with NPE for MVCC caches
Sergey Kozlov created IGNITE-10393:
--
Summary: DataStreamer failed with NPE for MVCC caches
Key: IGNITE-10393
URL: https://issues.apache.org/jira/browse/IGNITE-10393
Project: Ignite
Issue Type: Bug
Affects Versions: 2.7
Reporter: Sergey Kozlov
Fix For: 2.7
1. Start node with configured MVCC cache and PDS
2. Try to load data into cache by data streamer running on client node
{noformat}
OUT >>> datastreamer into cache 'cache_121', key=java.lang.Long,
value=org.apache.ignite.testtools.model.AllTypes, range=21..51
[14:55:16] (err) Failed to execute compound future reducer: GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=0, done=false, cancelled=false, err=null,
futs=TransformCollectionView [true, false, false, true]][14:55:16] (err) Failed
to execute compound future reducer: GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=0, done=false, cancelled=false, err=null,
futs=TransformCollectionView [true, true, false, true]][14:55:16] (err) Failed
to execute compound future reducer: GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=0, done=false, cancelled=false, err=null,
futs=TransformCollectionView [true, false, false, true]]class
org.apache.ignite.IgniteCheckedException: DataStreamer request failed
[node=5fae5c99-716e-441a-99d5-194de6ba61a6]
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onResponse(DataStreamerImpl.java:2055)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$3.onMessage(DataStreamerImpl.java:356)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Failed to write record:
MvccDataRecord [super=DataRecord [writeEntries=SingletonList [MvccDataEntry
[mvccVer=null]], super=TimeStampRecord [timestamp=1542974116042]]]
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.fillBuffer(FileWriteAheadLogManager.java:2634)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.addRecord(FileWriteAheadLogManager.java:2563)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle.access$1600(FileWriteAheadLogManager.java:2451)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:823)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:3414)
at
org.apache.ignite.internal.processors.cache.GridCacheEntryEx.initialValue(GridCacheEntryEx.java:766)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$IsolatedUpdater.receive(DataStreamerImpl.java:2265)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:400)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
... 7 more
Caused by: java.lang.NullPointerException
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.TxRecordSerializer.putMvccVersion(TxRecordSerializer.java:66)
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.putMvccDataEntry(RecordDataV2Serializer.java:298)
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.putPlainDataEntry(RecordDataV2Serializer.java:286)
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.writePlainRecord(RecordDataV2Serializer.java:246)
at
org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.writeRecord(RecordDataV1Serializer.java:222)
[jira] [Created] (IGNITE-10394) Try to activate cluster after deactivate. All node exit by handler
ARomantsov created IGNITE-10394: --- Summary: Try to activate cluster after deactivate. All node exit by handler Key: IGNITE-10394 URL: https://issues.apache.org/jira/browse/IGNITE-10394 Project: Ignite Issue Type: Bug Components: data structures Affects Versions: 2.7 Reporter: ARomantsov AE: ignite-sys-cache ..processors.cache.CacheRegistry.update(CacheRegistry.java:188) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] SomeFire commented on a change in pull request #78: IGNITE-10203 Support for alternative configurations for PR testing
SomeFire commented on a change in pull request #78: IGNITE-10203 Support for
alternative configurations for PR testing
URL: https://github.com/apache/ignite-teamcity-bot/pull/78#discussion_r235949008
##
File path:
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/teamcity/ignited/buildtype/ParametersCompacted.java
##
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.teamcity.ignited.buildtype;
+
+import com.google.common.base.MoreObjects;
+import com.google.common.base.Strings;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import org.apache.ignite.ci.tcmodel.conf.bt.Parameters;
+import org.apache.ignite.ci.tcmodel.conf.bt.Property;
+import org.apache.ignite.ci.teamcity.ignited.IStringCompactor;
+import org.apache.ignite.internal.util.GridIntList;
+
+public class ParametersCompacted {
+private GridIntList keys;
+private GridIntList values;
+
+/**
+ * Default constructor.
+ */
+public ParametersCompacted() {
+}
+
+/**
+ * @param compactor Compactor.
+ * @param ref Reference.
+ */
+public ParametersCompacted(IStringCompactor compactor, List ref)
{
+final int size = ref.size();
+keys = new GridIntList(size);
+values = new GridIntList(size);
+for (Property next : ref) {
+final String name = next.name();
+if (Strings.isNullOrEmpty(name))
+continue;
+
+final String value = next.getValue();
+if (Strings.isNullOrEmpty(value))
+continue;
+final int val = compactor.getStringId(value);
+final int stringId = compactor.getStringId(name);
+
+keys.add(stringId);
+values.add(val);
+}
+}
+
+public Parameters toParameters(IStringCompactor compactor) {
+List properties = null;
+
+if (keys.size() > 0) {
+properties = new ArrayList<>();
+
+final int size = keys.size();
+for (int i = 0; i < size; i++) {
+if (i >= values.size())
Review comment:
Move check inside `for` statement.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] SomeFire commented on a change in pull request #78: IGNITE-10203 Support for alternative configurations for PR testing
SomeFire commented on a change in pull request #78: IGNITE-10203 Support for
alternative configurations for PR testing
URL: https://github.com/apache/ignite-teamcity-bot/pull/78#discussion_r235949213
##
File path:
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/teamcity/ignited/buildtype/ParametersCompacted.java
##
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.teamcity.ignited.buildtype;
+
+import com.google.common.base.MoreObjects;
+import com.google.common.base.Strings;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+import org.apache.ignite.ci.tcmodel.conf.bt.Parameters;
+import org.apache.ignite.ci.tcmodel.conf.bt.Property;
+import org.apache.ignite.ci.teamcity.ignited.IStringCompactor;
+import org.apache.ignite.internal.util.GridIntList;
+
+public class ParametersCompacted {
+private GridIntList keys;
+private GridIntList values;
+
+/**
+ * Default constructor.
+ */
+public ParametersCompacted() {
+}
+
+/**
+ * @param compactor Compactor.
+ * @param ref Reference.
+ */
+public ParametersCompacted(IStringCompactor compactor, List ref)
{
+final int size = ref.size();
+keys = new GridIntList(size);
+values = new GridIntList(size);
+for (Property next : ref) {
+final String name = next.name();
+if (Strings.isNullOrEmpty(name))
+continue;
+
+final String value = next.getValue();
+if (Strings.isNullOrEmpty(value))
+continue;
+final int val = compactor.getStringId(value);
+final int stringId = compactor.getStringId(name);
+
+keys.add(stringId);
+values.add(val);
+}
+}
+
+public Parameters toParameters(IStringCompactor compactor) {
+List properties = null;
+
+if (keys.size() > 0) {
+properties = new ArrayList<>();
+
+final int size = keys.size();
+for (int i = 0; i < size; i++) {
+if (i >= values.size())
Review comment:
And empty line before `for`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] SomeFire commented on a change in pull request #78: IGNITE-10203 Support for alternative configurations for PR testing
SomeFire commented on a change in pull request #78: IGNITE-10203 Support for
alternative configurations for PR testing
URL: https://github.com/apache/ignite-teamcity-bot/pull/78#discussion_r235948717
##
File path:
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/teamcity/ignited/TeamcityIgnitedImpl.java
##
@@ -308,6 +330,120 @@ else if (midValStartDate.before(key))
return chains;
}
+/**
+ * Actualize saved list of composite suites for project.
+ *
+ * @param projectId Project id.
+ */
+private void actualizeSavedCompositeBuildTypesIds(String projectId) {
+if (projectId.equals(DEFAULT_PROJECT_ID)) {
+compositeBuildTypesIdsForDefaultProject =
+
fatBuildTypeDao.compositeBuildTypesIdsSortedByBuildNumberCounter(srvIdMaskHigh,
projectId);
+}
+}
+
+/** {@inheritDoc} */
+@Override public List
getCompositeBuildTypesIdsSortedByBuildNumberCounter(String projectId) {
+ensureActualizeBuildTypeRefsRequested();
+ensureActualizeBuildTypesRequested();
+
+return projectId.equals(DEFAULT_PROJECT_ID) ?
compositeBuildTypesIdsForDefaultProject :
+
fatBuildTypeDao.compositeBuildTypesIdsSortedByBuildNumberCounter(srvIdMaskHigh,
projectId);
+}
+
+/** {@inheritDoc} */
+@Override public List
getAllBuildTypesCompacted(String projectId) {
+ensureActualizeBuildTypeRefsRequested();
+
+return buildTypeRefDao.buildTypesCompacted(srvIdMaskHigh, projectId);
+}
+
+/**
+ * Ensure actualize BuildTypeRefs requested. Add this task to scheduler.
+ */
+private void ensureActualizeBuildTypeRefsRequested() {
+scheduler.sheduleNamed(taskName("actualizeAllBuildTypeRefs"),
+this::reindexBuildTypeRefs, 4, TimeUnit.HOURS);
+}
+
+/**
+ *Ensure actualize BuildTypes requested. Add this task to scheduler.
+ */
+private void ensureActualizeBuildTypesRequested() {
+scheduler.sheduleNamed(taskName("actualizeAllBuildTypes"),
+this::reindexBuildTypes, 24, TimeUnit.HOURS);
+}
+
+/**
+ * Re-index all references to "IgniteTests24Java8" suites.
+ */
+private void reindexBuildTypeRefs() {
+runActualizeBuildTypeRefs(DEFAULT_PROJECT_ID);
+}
+
+/**
+ * Re-index all "IgniteTests24Java8" suites.
+ */
+private void reindexBuildTypes() {
+runActualizeBuildTypes(DEFAULT_PROJECT_ID);
+}
+
+/**
+ * Re-index all project suites.
+ *
+ * @param projectId Project id.
+ * @return Statistics with the number of updated and requested buildTypes.
+ */
+@SuppressWarnings({"WeakerAccess", "UnusedReturnValue"})
+@MonitoredTask(name = "Reindex BuildTypes (projectId)", nameExtArgsIndexes
= {0})
+@AutoProfiling
+protected String runActualizeBuildTypes(String projectId) {
+List buildTypeIds =
buildTypeRefDao.buildTypeIds(srvIdMaskHigh, projectId);
+
+int updated = 0;
+
+for (String buildTypeId : buildTypeIds) {
+
+BuildType buildType = conn.getBuildType(buildTypeId);
+
+FatBuildTypeCompacted existingBuildType =
fatBuildTypeDao.getFatBuildType(srvIdMaskHigh, buildTypeId);
+
+if (fatBuildTypeDao.saveBuildType(srvIdMaskHigh, buildType,
existingBuildType) != null)
+updated++;
+}
+
+if (updated != 0)
+actualizeSavedCompositeBuildTypesIds(projectId);
+
+return "BuildTypes updated " + updated + " from " +
buildTypeIds.size() + " requested";
+}
+
+/**
+ * Re-index all references to project suites.
+ *
+ * @param projectId Project id.
+ * @return Statistics with the number of updated and requested
buildTypeRefs.
+ */
+@SuppressWarnings({"WeakerAccess", "UnusedReturnValue"})
+@MonitoredTask(name = "Reindex BuildTypeRefs (projectId)",
nameExtArgsIndexes = {0})
+@AutoProfiling
+protected String runActualizeBuildTypeRefs(String projectId) {
+List tcData = conn.getBuildTypes(projectId);
+
+Set buildsUpdated = buildTypeRefDao.saveChunk(srvIdMaskHigh,
tcData);
+
+Set removedBuildTypes =
buildTypeRefDao.markMissingBuildsAsRemoved(srvIdMaskHigh,
+
tcData.stream().map(BuildTypeRef::getId).collect(Collectors.toList()),
projectId);
+
+if (!(buildsUpdated.isEmpty() && removedBuildTypes.isEmpty())) {
Review comment:
Remove braces.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] SomeFire commented on a change in pull request #78: IGNITE-10203 Support for alternative configurations for PR testing
SomeFire commented on a change in pull request #78: IGNITE-10203 Support for
alternative configurations for PR testing
URL: https://github.com/apache/ignite-teamcity-bot/pull/78#discussion_r235915862
##
File path:
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
##
@@ -57,7 +54,12 @@
@Deprecated
long DEFAULT_BUILDS_COUNT = 1000;
-CompletableFuture> getProjectSuites(String projectId);
+/** {@inheritDoc} */
+@Override default List getBuildTypes(String projectId) {
+return FutureUtil.getResult(getProjectSuites(projectId));
+}
+
+CompletableFuture> getProjectSuites(String projectId);
Review comment:
Missed javadocs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ignite pull request #5434: IGNITE-10335 move ML examples datasets files to r...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5434 ---
[GitHub] ignite pull request #5489: Ignite 2.4.13.b1
GitHub user antkr opened a pull request: https://github.com/apache/ignite/pull/5489 Ignite 2.4.13.b1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-2.4.13.b1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5489.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5489 commit 247a9ef5a68de198106a866b277f1bee596c3633 Author: devozerov Date: 2018-04-02T10:16:00Z Merge branch 'ignite-2.4.4' into ignite-2.4-master commit 3949df02128e3003c409e0169ad0c80948ba134c Author: Aleksey Plekhanov Date: 2018-02-13T13:43:49Z IGNITE-5265 Eviction Rate memory metric to be implemented Signed-off-by: Anton Vinogradov commit 7bc6c1ad3bacaf5e354d2710931eb398c346c06c Author: Tim Onyschak Date: 2018-03-31T12:58:25Z IGNITE-7090 Semaphore Stuck when no acquirers to assign permit - Fixes #3443. Signed-off-by: dspavlov (cherry picked from commit 3fc5d57) commit c854ad86f4c64260171085c8c7cce97ba1ad2530 Author: Pavel Kovalenko Date: 2018-02-22T09:02:31Z IGNITE-7749 Fixed testDiscoCacheReuseOnNodeJoin test. - Fixes #3540. Signed-off-by: Alexey Goncharuk commit b2e94b36bddebbf7b3ff40631e099939d16926d2 Author: Alexey Goncharuk Date: 2018-02-13T15:53:23Z IGNITE-7692 Corrected test to not fail when SQL is executed on backup commit 852425d4170ed1871c79f5ac26bfb3d03cc36a6f Author: ÐлекÑей СÑелÑмак Date: 2018-04-06T15:28:22Z IGNITE-8049 Limit the number of operation cycles in B+Tree - Fixes #3769. Signed-off-by: dpavlov (cherry picked from commit e491f10) commit 42f529f0a04ce22786bb4a23032a64f93e214233 Author: Alexey Kuznetsov Date: 2018-04-09T02:25:50Z IGNITE-8159 control.sh: Fixed NPE on adding nodes on empty baseline and not active cluster. (cherry picked from commit 834869c) commit b5f180838246f895d36846ea707790c1ff7fe70a Author: Stanislav Lukyanov Date: 2018-04-09T11:33:13Z IGNITE-7904: Changed IgniteUtils::cast not to trim exception chains. This closes #3683. (cherry picked from commit 3a4f23b) commit 1ce8e1a8fd48469073592e2fb77e2881a168a219 Author: Roman Guseinov Date: 2018-04-09T11:45:44Z IGNITE-7944: Disconnected client node tries to send JOB_CANCEL message. Applied fix: - Skip sending message if client disconnected; - Throw IgniteCheckedException if a client node is disconnected and communication client is null. This closes #3737. (cherry picked from commit d70477b) commit 840e193b3e604e8b0d90be5fac16cf11d8c832e6 Author: Ilya Lantukh Date: 2018-04-06T10:49:10Z ignite-8087 : Backported ignite-8018. (cherry picked from commit 175edf3) commit 94e857c94ce5e7998fc39e38deee69f389ddbc5b Author: Alexey Goncharuk Date: 2018-04-10T17:33:47Z IGNITE-6430 Complete failing test early commit 770f7469d4b86ce9af61e9e43c7262d86ee05b54 Author: Eduard Shangareev Date: 2018-04-06T16:22:07Z IGNITE-8114 Add fail recovery mechanism to tracking pages commit 549d6e3de86a22697577f77a2ebab3d0dca7338d Author: Alexey Kukushkin Date: 2018-04-11T13:29:07Z IGNITE-8221: Security for thin clients. commit 94c0f56ef78e6c61f0e753c8aa0185c611630d45 Author: devozerov Date: 2018-04-11T13:44:33Z IGNITE-8148: JDBC thin: semicolon as delimiter for properties. This closes #3794. commit d99a50ccd6923aa48da78955207fe747b287ea81 Author: mcherkasov Date: 2018-04-11T00:23:29Z IGNITE-8153 Nodes fail to connect each other when SSL is enabled - Fixes #3773. Signed-off-by: Valentin Kulichenko commit 03285e953d005a18fb88a35d21701100f58f7a29 Author: devozerov Date: 2018-04-12T07:37:36Z IGNITE-8042: .NET thin client: authentication support. This closes #3790. commit 8e740164e8a0102a94f408d72b41f73df675cfb1 Author: Ilya Kasnacheev Date: 2018-04-05T09:55:36Z IGNITE-7712: SQL: system property to force lazy query execution. Cherry-picked from 064c816c177d31f18af2954175ca3ad0f3eee957 commit df405d439d461d3576cd3b22d53716cd324b3ef7 Author: Alexander Paschenko Date: 2018-04-11T14:03:16Z IGNITE-8204: SQL: fixed hangs when lazy flag is enabled. Cherry-picked from 747e6c5. commit 8af36584a81f6411a08b66935b843a5c1ba3169c Author: devozerov Date: 2018-04-12T12:02:57Z IGNITE-8135: SQL: authentication for CREATE TABLE and DROP TABLE commands. This closes #3801. commit 1c31e07bb7510b908b540169a6ded7a909e23fda Author: devozerov Date: 2018-04-12T12:13:51Z IGNITE-8230: SQL: Fixed backup number propagation in CREATE TABLE command. This closes #3803. commit 131b9b974c01d1b5c05f98e2fe3947f0672c94ec Author: tledkov-gridgain Date: 2018-04-16T08:28:39Z IGNITE-8129: MTCGA: setup default SSL context in
[GitHub] ignite pull request #5488: IGNITE-10392 : Use lasdAffChangedTopVer if DiscoC...
GitHub user ilantukh opened a pull request: https://github.com/apache/ignite/pull/5488 IGNITE-10392 : Use lasdAffChangedTopVer if DiscoCache for the requested topVer isn't ready. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10392 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5488.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5488 commit 22b39db36b820b4410be0e3b4f21d1b29ce760af Author: Ilya Lantukh Date: 2018-11-23T13:36:44Z IGNITE-10392 : Use lasdAffChangedTopVer if DiscoCache for the requested topVer isn't ready. ---
[GitHub] ignite pull request #5487: IGNITE-10385 replace empty fields with zero-lengt...
GitHub user antkr opened a pull request: https://github.com/apache/ignite/pull/5487 IGNITE-10385 replace empty fields with zero-length arrays. TC run. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10385 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5487.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5487 commit 949dc3daf9e980b357ccb6e1ebc8cf7eb0f5e391 Author: antkr Date: 2018-11-23T13:34:21Z IGNITE-10385 replace empty fields with zero-length arrays. ---
[jira] [Created] (IGNITE-10392) Client broken cluster where try to connect. Server nodes drop by handler
Ilya Lantukh created IGNITE-10392:
-
Summary: Client broken cluster where try to connect. Server nodes
drop by handler
Key: IGNITE-10392
URL: https://issues.apache.org/jira/browse/IGNITE-10392
Project: Ignite
Issue Type: Bug
Reporter: Ilya Lantukh
Assignee: Ilya Lantukh
{noformat}
org.apache.ignite.IgniteException: Failed to resolve nodes topology
[cacheGrp=N/A, topVer=AffinityTopologyVersion [topVer=133, minorTopVer=0],
history=[AffinityTopologyVersion [topVer=35, minorTopVer=0],
AffinityTopologyVersion [topVer=36, minorTopVer=0], AffinityTopologyVersion
[topVer=37, minorTopVer=0], AffinityTopologyVersion [topVer=38, minorTopVer=0],
AffinityTopologyVersion [topVer=39, minorTopVer=0], AffinityTopologyVersion
[topVer=40, minorTopVer=0], AffinityTopologyVersion [topVer=41, minorTopVer=0],
AffinityTopologyVersion [topVer=42, minorTopVer=0], AffinityTopologyVersion
[topVer=43, minorTopVer=0], AffinityTopologyVersion [topVer=44, minorTopVer=0],
AffinityTopologyVersion [topVer=45, minorTopVer=0], AffinityTopologyVersion
[topVer=46, minorTopVer=0], AffinityTopologyVersion [topVer=47, minorTopVer=0],
AffinityTopologyVersion [topVer=48, minorTopVer=0], AffinityTopologyVersion
[topVer=49, minorTopVer=0], AffinityTopologyVersion [topVer=50, minorTopVer=0],
AffinityTopologyVersion [topVer=51, minorTopVer=0], AffinityTopologyVersion
[topVer=52, minorTopVer=0], AffinityTopologyVersion [topVer=53, minorTopVer=0],
AffinityTopologyVersion [topVer=54, minorTopVer=0], AffinityTopologyVersion
[topVer=55, minorTopVer=0], AffinityTopologyVersion [topVer=56, minorTopVer=0],
AffinityTopologyVersion [topVer=57, minorTopVer=0], AffinityTopologyVersion
[topVer=58, minorTopVer=0], AffinityTopologyVersion [topVer=59, minorTopVer=0],
AffinityTopologyVersion [topVer=60, minorTopVer=0], AffinityTopologyVersion
[topVer=61, minorTopVer=0], AffinityTopologyVersion [topVer=62, minorTopVer=0],
AffinityTopologyVersion [topVer=63, minorTopVer=0], AffinityTopologyVersion
[topVer=64, minorTopVer=0], AffinityTopologyVersion [topVer=65, minorTopVer=0],
AffinityTopologyVersion [topVer=66, minorTopVer=0], AffinityTopologyVersion
[topVer=67, minorTopVer=0], AffinityTopologyVersion [topVer=68, minorTopVer=0],
AffinityTopologyVersion [topVer=69, minorTopVer=0], AffinityTopologyVersion
[topVer=70, minorTopVer=0], AffinityTopologyVersion [topVer=71, minorTopVer=0],
AffinityTopologyVersion [topVer=72, minorTopVer=0], AffinityTopologyVersion
[topVer=73, minorTopVer=0], AffinityTopologyVersion [topVer=74, minorTopVer=0],
AffinityTopologyVersion [topVer=75, minorTopVer=0], AffinityTopologyVersion
[topVer=76, minorTopVer=0], AffinityTopologyVersion [topVer=77, minorTopVer=0],
AffinityTopologyVersion [topVer=78, minorTopVer=0], AffinityTopologyVersion
[topVer=79, minorTopVer=0], AffinityTopologyVersion [topVer=80, minorTopVer=0],
AffinityTopologyVersion [topVer=81, minorTopVer=0], AffinityTopologyVersion
[topVer=82, minorTopVer=0], AffinityTopologyVersion [topVer=83, minorTopVer=0],
AffinityTopologyVersion [topVer=84, minorTopVer=0], AffinityTopologyVersion
[topVer=85, minorTopVer=0], AffinityTopologyVersion [topVer=86, minorTopVer=0],
AffinityTopologyVersion [topVer=87, minorTopVer=0], AffinityTopologyVersion
[topVer=88, minorTopVer=0], AffinityTopologyVersion [topVer=89, minorTopVer=0],
AffinityTopologyVersion [topVer=90, minorTopVer=0], AffinityTopologyVersion
[topVer=91, minorTopVer=0], AffinityTopologyVersion [topVer=92, minorTopVer=0],
AffinityTopologyVersion [topVer=93, minorTopVer=0], AffinityTopologyVersion
[topVer=94, minorTopVer=0], AffinityTopologyVersion [topVer=95, minorTopVer=0],
AffinityTopologyVersion [topVer=96, minorTopVer=0], AffinityTopologyVersion
[topVer=97, minorTopVer=0], AffinityTopologyVersion [topVer=98, minorTopVer=0],
AffinityTopologyVersion [topVer=99, minorTopVer=0], AffinityTopologyVersion
[topVer=100, minorTopVer=0], AffinityTopologyVersion [topVer=101,
minorTopVer=0], AffinityTopologyVersion [topVer=102, minorTopVer=0],
AffinityTopologyVersion [topVer=103, minorTopVer=0], AffinityTopologyVersion
[topVer=104, minorTopVer=0], AffinityTopologyVersion [topVer=105,
minorTopVer=0], AffinityTopologyVersion [topVer=106, minorTopVer=0],
AffinityTopologyVersion [topVer=107, minorTopVer=0], AffinityTopologyVersion
[topVer=108, minorTopVer=0], AffinityTopologyVersion [topVer=109,
minorTopVer=0], AffinityTopologyVersion [topVer=110, minorTopVer=0],
AffinityTopologyVersion [topVer=111, minorTopVer=0], AffinityTopologyVersion
[topVer=112, minorTopVer=0], AffinityTopologyVersion [topVer=113,
minorTopVer=0], AffinityTopologyVersion [topVer=114, minorTopVer=0],
AffinityTopologyVersion [topVer=115, minorTopVer=0], AffinityTopologyVersion
[topVer=116, minorTopVer=0], AffinityTopologyVersion [topVer=117,
minorTopVer=0], AffinityTopologyVersion [topVer=118,
Re: [DISCUSSION] Design document. Rebalance caches by transferring partition files
Hello! This proposal will also happily break my compression-with-dictionary patch since it relies currently on only having local dictionaries. However, when you have compressed data, maybe speed boost is even greater with your approach. Regards, -- Ilya Kasnacheev пт, 23 нояб. 2018 г. в 13:08, Maxim Muzafarov : > Igniters, > > > I'd like to take the next step of increasing the Apache Ignite with > enabled persistence rebalance speed. Currently, the rebalancing > procedure doesn't utilize the network and storage device throughout to > its full extent even with enough meaningful values of > rebalanceThreadPoolSize property. As part of the previous discussion > `How to make rebalance faster` [1] and IEP-16 [2] Ilya proposed an > idea [3] of transferring cache partition files over the network. > From my point, the case to which this type of rebalancing procedure > can bring the most benefit – is adding a completely new node or set of > new nodes to the cluster. Such a scenario implies fully relocation of > cache partition files to the new node. To roughly estimate the > superiority of partition file transmitting over the network the native > Linux scp\rsync commands can be used. My test environment showed the > result of the new approach as 270 MB/s vs the current 40 MB/s > single-threaded rebalance speed. > > > I've prepared the design document IEP-28 [4] and accumulated all the > process details of a new rebalance approach on that page. Below you > can find the most significant details of the new rebalance procedure > and components of the Apache Ignite which are proposed to change. > > Any feedback is very appreciated. > > > *PROCESS OVERVIEW* > > The whole process is described in terms of rebalancing single cache > group and partition files would be rebalanced one-by-one: > > 1. The demander node sends the GridDhtPartitionDemandMessage to the > supplier node; > 2. When the supplier node receives GridDhtPartitionDemandMessage and > starts the new checkpoint process; > 3. The supplier node creates empty the temporary cache partition file > with .tmp postfix in the same cache persistence directory; > 4. The supplier node splits the whole cache partition file into > virtual chunks of predefined size (multiply to the PageMemory size); > 4.1. If the concurrent checkpoint thread determines the appropriate > cache partition file chunk and tries to flush dirty page to the cache > partition file > 4.1.1. If rebalance chunk already transferred > 4.1.1.1. Flush the dirty page to the file; > 4.1.2. If rebalance chunk not transferred > 4.1.2.1. Write this chunk to the temporary cache partition file; > 4.1.2.2. Flush the dirty page to the file; > 4.2. The node starts sending to the demander node each cache partition > file chunk one by one using FileChannel#transferTo > 4.2.1. If the current chunk was modified by checkpoint thread – read > it from the temporary cache partition file; > 4.2.2. If the current chunk is not touched – read it from the original > cache partition file; > 5. The demander node starts to listen to new pipe incoming connections > from the supplier node on TcpCommunicationSpi; > 6. The demander node creates the temporary cache partition file with > .tmp postfix in the same cache persistence directory; > 7. The demander node receives each cache partition file chunk one by one > 7.1. The node checks CRC for each PageMemory in the downloaded chunk; > 7.2. The node flushes the downloaded chunk at the appropriate cache > partition file position; > 8. When the demander node receives the whole cache partition file > 8.1. The node initializes received .tmp file as its appropriate cache > partition file; > 8.2. Thread-per-partition begins to apply for data entries from the > beginning of WAL-temporary storage; > 8.3. All async operations corresponding to this partition file still > write to the end of temporary WAL; > 8.4. At the moment of WAL-temporary storage is ready to be empty > 8.4.1. Start the first checkpoint; > 8.4.2. Wait for the first checkpoint ends and own the cache partition; > 8.4.3. All operations now are switched to the partition file instead > of writing to the temporary WAL; > 8.4.4. Schedule the temporary WAL storage deletion; > 9. The supplier node deletes the temporary cache partition file; > > > *COMPONENTS TO CHANGE* > > CommunicationSpi > > To benefit from zero copy we must delegate the file transferring to > FileChannel#transferTo(long, long, > java.nio.channels.WritableByteChannel) because the fast path of > transferTo method is only executed if the destination buffer inherits > from an internal JDK class. > > Preloader > > A new implementation of cache entries preloader assume to be done. The > new implementation must send and receive cache partition files over > the CommunicationSpi channels by chunks of data with validation > received items. The new layer over the cache partition file must > support direct using of FileChannel#transferTo method over the > CommunicationSpi
[GitHub] asfgit closed pull request #81: IGNITE-10372 Optimize master trends page step 3: Tests migrated to new concept
asfgit closed pull request #81: IGNITE-10372 Optimize master trends page step 3: Tests migrated to new concept URL: https://github.com/apache/ignite-teamcity-bot/pull/81 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: Improve lazy mode SQL query execution (IGNITE-9171)
I would say that we have multiple conditions which may require re-try from user side - MVCC write conflict, lock timeout, node crash. Now concurrent DDL operation will be added to the list. I would think about proper retry exception semantics in a separate task. On Thu, Nov 22, 2018 at 5:37 PM Taras Ledkov wrote: > Hi, > > Lets discuss changes of Ignite public interface relates to retry SQL > queries. > > Should we add new public exception (.e.g: `QueryRetryException`) for > cache SQL API > and special vendor error code to SQLException for JDBC? > > 21.11.2018 16:32, Vladimir Ozerov пишет: > > Hi Taras, > > > > Unfortunately, this will not be easy to implement. We already have > > AffinityTopologyVersion which is updated on certain schema change > > operations, such as CREATE/DROP TABLE. Moreover, it is already passed in > > query request messages (GridH2QueryRequest.topVer). Unfortunately, it is > > not updated for other DDL scenarios, such as ALTER TABLE. We can > introduce > > our own global counter. We can try to integrate with > AffinityTopologyVersion. > > Both solutions will require tight integration with PME mechanism to > > function properly. But good news is that this is not a new problem. > > Currently it is already possible for two map requests to be executed on > > different schemas: > > > > client_1: SEND_QUERY > > client_2: SEND_ALTER > > node_1 :EXEC_QUERY EXEC_ALTER > > node_2 : EXEC_ALTER EXEC_QUERY > > > > So I would say that nothing changes for us even after proposed changes, > and > > way may leave this problem aside for now. > > > > Makes sense? > > > > On Wed, Nov 21, 2018 at 4:15 PM Taras Ledkov > wrote: > > > >> Vladimir, > >> > >> The query protocol will be changed in both solution. > >> > >> The tables versions must be added to the 'GridQueryNextPageResponse' > >> message > >> and must be compared on the reduce node too. Because the DLL / DML race > >> may be happens on different nodes. > >> > >> 21.11.2018 15:24, Vladimir Ozerov пишет: > >>> Taras, > >>> > >>> Thank you for analysis. I'd like to add that there is another solution > - > >>> PME :-) In this case DDL will be blocked until current SELECTs and > >> updates > >>> are finished, and do not allow new operations to be executed. This > >> approach > >>> solves all issues - it avoids deadlocks between query threads and DDL > >>> threads when they are reordered on different nodes, it avoids > starvation > >> of > >>> DDL operations, and it doesn't cancel any queries. But there is serious > >>> drawback - performance. The drawback is that it is more complex to > >>> implement (query protocol changes might be required), it blocks the > >> cluster > >>> even when it is needed, and it may destabilize PME mechanism, which is > >>> already on his last legs. > >>> > >>> For this reason killing queries which interleave with DDL looks like a > >>> balanced solution for now - it is reasonably simple, allows us to we > >> avoid > >>> OOME in many cases, and do not introduce any additional complexity for > >>> users, as they are already prepared for re-tries. > >>> > >>> But I would like to stress one thing - we will need integration with > PME > >> at > >>> some point in time anyway. Some DDL operations are blocking in their > >> nature > >>> (e.g. DROP COLUMN). Other DDL operations may be non-blocking, but > >> blocking > >>> implementation may give them serious performance benefits (e.g. CREATE > >>> INDEX). > >>> > >>> So I propose to go with your solution for now, and start thinking about > >> SQL > >>> -> PME integration in the background. > >>> > >>> Vladimir. > >>> > >>> On Wed, Nov 21, 2018 at 2:23 PM Taras Ledkov > >> wrote: > Hi community, > > We will enhance lazy mode for SQL query execution. > > Lazy mode overview: > Lazy mode is related to H2 lazy mode when the all query results are > not > copied to the RAM in some cases. > The mode it is applicable for SELECTs that doesn't not require > materialize all results in memory, e.g. simple scan plans, IDX > lookup, > merge join etc. > And not applicable for SORT by not indexed fields, aggregates, nested > loops joins etc. > > When mode is applicable it produces result with iterator-like behavior > on server side and not consume RAM. > So the huge result set may be selected without OOME. > > The current implementation. > The current implementation is start separate thread for each query > with > 'lazy=true'. > This is caused by the our implementation of 'GridH2Table'. In details: > the table's locks. > The table must be locked while result set is completed. > > When lazy is disabled a complete result is generated on the first step > of a query execution (then tables unlock) > and result is stored on the node and sent to other node (or client) > page > by page. > > When lazy is enabled
Re: [IMPORTANT] Future of Binary Objects
Ok, let's agree on the fact that we would like to make schema change rules less restrictive. But how less - is separate topic. Use case which annoys me the most is DROP/ADD COLUMN commands. On Thu, Nov 22, 2018 at 12:25 PM Sergi Vladykin wrote: > If we are developing a product for users, we already guessing what is right > and what is wrong for them. So let's avoid these sophistic statements. > > In the end it is always our responsibility to provide a balanced set of > trade-offs between > usability, performance and safety. > > Let me repeat, I'm not against any possible type conversions, but I'm > strongly against binary incompatible ones. > If we always store List.of(1) as 1 and make them binary interchangeable, > I'm OK with that. > > And still for good practices I'd suggest to look at what Protobuf allows > and what not: > https://developers.google.com/protocol-buffers/docs/proto3#updating > > Sergi > > чт, 22 нояб. 2018 г. в 11:04, Vladimir Ozerov : > > > Sergi, > > > > I think we should not guess for users what is right or wrong for them. It > > is up to user to decide what is valid. For example, consider a user who > > operates on a list of Integers, and to optimize memory consumption he > > decide to save in the same field either List, or plain Integer > in > > case only single element exists. Another example - a kind of data lake or > > data cleansing application, which may receive the same field in different > > forms. E.g. age in the form of Integer or String. Does it work for user > or > > not? We do not know. Will he need to migrate the whole data set? We do > not > > know either. > > > > The only place in the product where we case is SQL. But in this case > > instead of adding checks on binary level, we should validate data on > cache > > level. In fact, Ignite already works this way. E.g. nullability checks > are > > performed on cache level rather than binary. All we need is to move all > > checks to cache level from binary level. > > > > > > On Thu, Nov 22, 2018 at 9:41 AM Sergi Vladykin > > > wrote: > > > > > It may be OK to extend compatible field types (like from Int to Long). > > > > > > In Protobuf for example this is allowed just because there is no > > difference > > > between Int and Long in binary format: they all are equally varlen > > encoded > > > and Longs just will occupy up to 9 bytes, while Ints up to 5. > > > > > > But for every other case, where binary representation is type > dependent, > > I > > > would be against. This will either require to migrate the whole dataset > > to > > > a new model (which is always risky, since you may need to rollback to > > > previous version of your code) or it will require type > checks/conversions > > > for each field access, which is a hard to reason complication and > > possible > > > performance penalty. > > > > > > Sergi > > > > > > > > > > > > чт, 22 нояб. 2018 г. в 09:23, Vladimir Ozerov : > > > > > > > Denis, > > > > > > > > Several examples: > > > > 1) DEFAULT values - in SQL you may avoid storing default value in the > > > table > > > > and store it in metadata instead. Not applicable for BinaryObject > > because > > > > the same binary object may be saved to two SQL tables with different > > > > defaults > > > > 2) DATE and other temporal types - in SQL you want to store it in > > special > > > > format to be able to extract date parts quickly (typically - 11 > bytes). > > > But > > > > in Java and some other languages the best format is plain long. this > is > > > why > > > > we use it BinaryObject > > > > 3) String charset - in SQL you may choose different charsets for > > > different > > > > tables. E.g. UTF-8 for one, ASCII for another. In BinaryObject we > store > > > > everything in UTF-8, and this is fine for most cases, well ... except > > of > > > > SQL :-) > > > > > > > > The key thing here is that you cannot define a format which will be > > good > > > > for both SQL, and native API. They are very different. This is why I > > > > propose to define additional interface on cache level defining how to > > > store > > > > values, which will be very different from binary objects. > > > > > > > > Vladimir. > > > > > > > > On Thu, Nov 22, 2018 at 3:32 AM Denis Magda > wrote: > > > > > > > > > Vladimir, > > > > > > > > > > Could you educate me a little bit, why the current format is bad > for > > > SQL > > > > > and why another one is more suitable? > > > > > > > > > > Also, if we introduce the new format then why would we keep the > > binary > > > > one? > > > > > Is the new format just a next version of the binary one. > > > > > > > > > > 2.3) Remove restrictions on changing field type > > > > > > I do not know why we did that in the first place. This > restriction > > > > > prevents > > > > > > type evolution and confuses users. > > > > > > > > > > > > > > > That is a hot requirement shared by those who use Ignite SQL in > > > > production. > > > > > +1. > > > > > > > > > > -- > > > > > Denis > > > > > > > > > > On
Re: Rename "cache group" to "tablespace"?
Hi Ivan,
I do not have answers to some of your questions, as haven't designed
tablespaces for Ignite so far. This is common term used by many vendors
(Oracle, MySQL, MongoDB to name a few). Essentially, "tablespace" is how
data of certain objects are organized into files. Multiple tablespaces for
separate objects provides performance and management benefits. For example,
different sets of objects might be written to different disks on local
machine to improve throughput. Or writes to a single object (e.g. cache)
may be parallelized in the same way.
But in this topic I would prefer to avoid digging much into details. My
goal for now is to understand whether we can create a single concept which
will be applicable to both existing and future storage formats.
Vladimir.
On Fri, Nov 23, 2018 at 10:41 AM Павлухин Иван wrote:
> Hi Vladimir,
>
> How do you see a default tablespace configuration in the "bright"
> future? Is it going to be a single file for an Ignite instance? In
> what cases will be there benefits to explicitly configure additional
> tablespaces?
> By the way was the name "tablespace" hand-crafted or it was it
> borrowed somewhere?
> Also if it is TABLEspace should it be aligned with renaming
> IgniteCache to IgniteTable discussed in [1]?
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/Applicability-of-term-cache-to-Apache-Ignite-td36541.html
> вт, 20 нояб. 2018 г. в 10:57, Dmitriy Setrakyan :
> >
> > I really like the idea.
> >
> > On Mon, Nov 19, 2018, 23:45 Vladimir Ozerov >
> > > Igniters,
> > >
> > > We had several discussion about cache groups [1]. Our current
> consensus is
> > > that this is not that good design decision - bad scan performance, no
> API.
> > > A kind of shortcut to solve memory consumption problems. For this
> reason we
> > > try to avoid exposing "cache group" to API when possible. Also we
> > > understand that current file-per-partition approach is not ideal
> either -
> > > when there are too many partitions we have to fsync a lot of files.
> And as
> > > an idea for "bright future" we consider tablespaces - one or several
> files
> > > where all database objects are stored in dedicated "segments", which
> are
> > > allocated in smaller units called "extents".
> > >
> > > I was thinking on how to "legalize" cache groups on API on the one hand
> > > (product needs them currently), and to let real tablespaces to appear
> in
> > > future. Idea is simple - treat cache group as special kind of
> tablespace.
> > > We may say that current storage organization is one type of
> tablespace, and
> > > proposed "file-segment-extent" storage organization is another type of
> > > tablespace.
> > >
> > > With this in mind we can configure tablespaces in IgniteConfiguration,
> then
> > > assign each cache a tablespace. As advanced tasks we may allow dynamic
> > > table space create/alter/drop, I'll show SQL examples below.
> > >
> > > // Interface
> > > interface Tablespace {
> > > String name();
> > > }
> > >
> > > // Current non-shared tablespace (aka "physical cache")
> > > class FilePerPartitionTablespace implements Tablespace {
> > > // ...
> > > }
> > >
> > > // Shared tablespace (aka "cache group") - note that now we have a
> legal
> > > place for cache group configuration:
> > > class FilePerPartitionSharedTablespace implements Tablespace {
> > > CacheMode cacheMode;
> > > CacheAtomicityMode cacheAtomicityMode;
> > > AffinityFunction affinityFunction;
> > > // + Other parameters from
> > > ClusterCachesInfo.validateCacheGroupConfiguration
> > > // + Some parameters from "DataRegionConfiguration" (e.g. paths)
> > > }
> > >
> > > // Future "real" tablespace
> > > class SegmentedTablespace implements Tablespace {
> > > // Whatever is needed.
> > > }
> > >
> > > // Cache configuration
> > > class CacheConfiguration {
> > > ...
> > > String tablespace;
> > >...
> > > }
> > >
> > > // Managing tablespaces from SQL:
> > > CREATE TABLESPACE my_tbs ENGINE=FILE_PER_PARTITION_SHARED
> > > CACHE_MODE=PARTITIONED BACKUPS=1
> > > ALTER TABLESPACE my_tbs ENCRYPTED=1
> > > CREATE TABLE my_table (...) TABLESPACE my_tbs
> > >
> > > What do you think?
> > >
> > > Vladimir.
> > >
> > > [1]
> > >
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/Remove-cache-groups-in-AI-3-0-td29184.html
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>
[GitHub] ignite pull request #5476: IGNITE-10323 Contol utility --deactivate on non-a...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5476 ---
Re: [DISCUSSION] Design document. Rebalance caches by transferring partition files
Maxim, Do we have any analysis on where time is really spent during current rebalance implementation? Proposed change may work very well. But it is rather complex, doesn't help in-memory mode, and will not work for other storage formats which we may potentially have in future (e.g. tablespace approach). Another thing to consider is MVCC - under load partitions may have a lot of entries which are not visible to anyone and is about to be removed, so there is no need to transfer them at all. Did we investigate any universal and less intrusive approaches to rebalance speedup before that? For example: - batched data block reads on supplier - iteration over partition rather than cache data tree on supplier - batched data save on demander - delayed free list and index re-build in demander Vladimir. On Fri, Nov 23, 2018 at 1:08 PM Maxim Muzafarov wrote: > Igniters, > > > I'd like to take the next step of increasing the Apache Ignite with > enabled persistence rebalance speed. Currently, the rebalancing > procedure doesn't utilize the network and storage device throughout to > its full extent even with enough meaningful values of > rebalanceThreadPoolSize property. As part of the previous discussion > `How to make rebalance faster` [1] and IEP-16 [2] Ilya proposed an > idea [3] of transferring cache partition files over the network. > From my point, the case to which this type of rebalancing procedure > can bring the most benefit – is adding a completely new node or set of > new nodes to the cluster. Such a scenario implies fully relocation of > cache partition files to the new node. To roughly estimate the > superiority of partition file transmitting over the network the native > Linux scp\rsync commands can be used. My test environment showed the > result of the new approach as 270 MB/s vs the current 40 MB/s > single-threaded rebalance speed. > > > I've prepared the design document IEP-28 [4] and accumulated all the > process details of a new rebalance approach on that page. Below you > can find the most significant details of the new rebalance procedure > and components of the Apache Ignite which are proposed to change. > > Any feedback is very appreciated. > > > *PROCESS OVERVIEW* > > The whole process is described in terms of rebalancing single cache > group and partition files would be rebalanced one-by-one: > > 1. The demander node sends the GridDhtPartitionDemandMessage to the > supplier node; > 2. When the supplier node receives GridDhtPartitionDemandMessage and > starts the new checkpoint process; > 3. The supplier node creates empty the temporary cache partition file > with .tmp postfix in the same cache persistence directory; > 4. The supplier node splits the whole cache partition file into > virtual chunks of predefined size (multiply to the PageMemory size); > 4.1. If the concurrent checkpoint thread determines the appropriate > cache partition file chunk and tries to flush dirty page to the cache > partition file > 4.1.1. If rebalance chunk already transferred > 4.1.1.1. Flush the dirty page to the file; > 4.1.2. If rebalance chunk not transferred > 4.1.2.1. Write this chunk to the temporary cache partition file; > 4.1.2.2. Flush the dirty page to the file; > 4.2. The node starts sending to the demander node each cache partition > file chunk one by one using FileChannel#transferTo > 4.2.1. If the current chunk was modified by checkpoint thread – read > it from the temporary cache partition file; > 4.2.2. If the current chunk is not touched – read it from the original > cache partition file; > 5. The demander node starts to listen to new pipe incoming connections > from the supplier node on TcpCommunicationSpi; > 6. The demander node creates the temporary cache partition file with > .tmp postfix in the same cache persistence directory; > 7. The demander node receives each cache partition file chunk one by one > 7.1. The node checks CRC for each PageMemory in the downloaded chunk; > 7.2. The node flushes the downloaded chunk at the appropriate cache > partition file position; > 8. When the demander node receives the whole cache partition file > 8.1. The node initializes received .tmp file as its appropriate cache > partition file; > 8.2. Thread-per-partition begins to apply for data entries from the > beginning of WAL-temporary storage; > 8.3. All async operations corresponding to this partition file still > write to the end of temporary WAL; > 8.4. At the moment of WAL-temporary storage is ready to be empty > 8.4.1. Start the first checkpoint; > 8.4.2. Wait for the first checkpoint ends and own the cache partition; > 8.4.3. All operations now are switched to the partition file instead > of writing to the temporary WAL; > 8.4.4. Schedule the temporary WAL storage deletion; > 9. The supplier node deletes the temporary cache partition file; > > > *COMPONENTS TO CHANGE* > > CommunicationSpi > > To benefit from zero copy we must delegate the file transferring to > FileChannel#transferTo(long, long,
[GitHub] ignite pull request #5438: Tde with fixes
Github user nizhikov closed the pull request at: https://github.com/apache/ignite/pull/5438 ---
[jira] [Created] (IGNITE-10391) MVCC: Invoke request fails on backup while rebalance is in progress.
Andrew Mashenkov created IGNITE-10391: - Summary: MVCC: Invoke request fails on backup while rebalance is in progress. Key: IGNITE-10391 URL: https://issues.apache.org/jira/browse/IGNITE-10391 Project: Ignite Issue Type: Bug Components: mvcc Reporter: Andrew Mashenkov Fix For: 2.8 Invoke request fails with Assertion on backup while rebalance is in progress. Enlist request handler expects entry processor instead of value in case of Invoke operation, but when rebalance is in progress we pass entry history to backup side. This causes assertion triggering. We have to handle correctly this case and apply history and then entry processor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #5486: IGNITE-10390 Fixed BPlusTree#isEmpty
GitHub user agoncharuk opened a pull request: https://github.com/apache/ignite/pull/5486 IGNITE-10390 Fixed BPlusTree#isEmpty You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10390 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5486.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5486 commit fc8ee6b42785d4da11d19df26d2314b43ea53845 Author: Alexey Goncharuk Date: 2018-11-23T10:07:42Z IGNITE-10390 Fixed BPlusTree#isEmpty ---
[DISCUSSION] Design document. Rebalance caches by transferring partition files
Igniters, I'd like to take the next step of increasing the Apache Ignite with enabled persistence rebalance speed. Currently, the rebalancing procedure doesn't utilize the network and storage device throughout to its full extent even with enough meaningful values of rebalanceThreadPoolSize property. As part of the previous discussion `How to make rebalance faster` [1] and IEP-16 [2] Ilya proposed an idea [3] of transferring cache partition files over the network. >From my point, the case to which this type of rebalancing procedure can bring the most benefit – is adding a completely new node or set of new nodes to the cluster. Such a scenario implies fully relocation of cache partition files to the new node. To roughly estimate the superiority of partition file transmitting over the network the native Linux scp\rsync commands can be used. My test environment showed the result of the new approach as 270 MB/s vs the current 40 MB/s single-threaded rebalance speed. I've prepared the design document IEP-28 [4] and accumulated all the process details of a new rebalance approach on that page. Below you can find the most significant details of the new rebalance procedure and components of the Apache Ignite which are proposed to change. Any feedback is very appreciated. *PROCESS OVERVIEW* The whole process is described in terms of rebalancing single cache group and partition files would be rebalanced one-by-one: 1. The demander node sends the GridDhtPartitionDemandMessage to the supplier node; 2. When the supplier node receives GridDhtPartitionDemandMessage and starts the new checkpoint process; 3. The supplier node creates empty the temporary cache partition file with .tmp postfix in the same cache persistence directory; 4. The supplier node splits the whole cache partition file into virtual chunks of predefined size (multiply to the PageMemory size); 4.1. If the concurrent checkpoint thread determines the appropriate cache partition file chunk and tries to flush dirty page to the cache partition file 4.1.1. If rebalance chunk already transferred 4.1.1.1. Flush the dirty page to the file; 4.1.2. If rebalance chunk not transferred 4.1.2.1. Write this chunk to the temporary cache partition file; 4.1.2.2. Flush the dirty page to the file; 4.2. The node starts sending to the demander node each cache partition file chunk one by one using FileChannel#transferTo 4.2.1. If the current chunk was modified by checkpoint thread – read it from the temporary cache partition file; 4.2.2. If the current chunk is not touched – read it from the original cache partition file; 5. The demander node starts to listen to new pipe incoming connections from the supplier node on TcpCommunicationSpi; 6. The demander node creates the temporary cache partition file with .tmp postfix in the same cache persistence directory; 7. The demander node receives each cache partition file chunk one by one 7.1. The node checks CRC for each PageMemory in the downloaded chunk; 7.2. The node flushes the downloaded chunk at the appropriate cache partition file position; 8. When the demander node receives the whole cache partition file 8.1. The node initializes received .tmp file as its appropriate cache partition file; 8.2. Thread-per-partition begins to apply for data entries from the beginning of WAL-temporary storage; 8.3. All async operations corresponding to this partition file still write to the end of temporary WAL; 8.4. At the moment of WAL-temporary storage is ready to be empty 8.4.1. Start the first checkpoint; 8.4.2. Wait for the first checkpoint ends and own the cache partition; 8.4.3. All operations now are switched to the partition file instead of writing to the temporary WAL; 8.4.4. Schedule the temporary WAL storage deletion; 9. The supplier node deletes the temporary cache partition file; *COMPONENTS TO CHANGE* CommunicationSpi To benefit from zero copy we must delegate the file transferring to FileChannel#transferTo(long, long, java.nio.channels.WritableByteChannel) because the fast path of transferTo method is only executed if the destination buffer inherits from an internal JDK class. Preloader A new implementation of cache entries preloader assume to be done. The new implementation must send and receive cache partition files over the CommunicationSpi channels by chunks of data with validation received items. The new layer over the cache partition file must support direct using of FileChannel#transferTo method over the CommunicationSpi pipe connection. The connection bandwidth of the cache partition file transfer must have the ability to be limited at runtime. Checkpointer When the supplier node receives the cache partition file demand request it will send the file over the CommunicationSpi. The cache partition file can be concurrently updated by checkpoint thread during its transmission. To guarantee the file consistency Сheckpointer must use copy-on-write technique and save a copy of updated chunk into the temporary
[jira] [Created] (IGNITE-10390) BPlusTree#isEmpty() does not release root page
Alexey Goncharuk created IGNITE-10390: - Summary: BPlusTree#isEmpty() does not release root page Key: IGNITE-10390 URL: https://issues.apache.org/jira/browse/IGNITE-10390 Project: Ignite Issue Type: Bug Reporter: Alexey Goncharuk -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Brainstorm: Make TC Run All faster
Hi, I have some thoughts about tests speed. First of all I must say that I do not see any simple and lightweight solution. I did some measurements a while ago and it looks like that simply optimizing number of Ignite node start/stop calls will not give us great speed up. I naively measured [1] time spent in node start/stop methods and for Cache 1 suite it turns out that only 2.5 minutes is spent there when whole suite run is about 1 hour. But I made an experiment only for Cache 1 suite and might have been not accurate enough. Moreover I thought about refactoring whole test base to employ some faster patterns and it looks unfeasible to me to accomplish with it in relatively short amount of time. So, I can imagine a following approach. There are a lot of load/concurrent tests, which executions are limited either by big number of iterations (1000+) or by time (30 sec, 60 sec). In ideal world I think there should not be such tests in Run All. If such tests regularly catch bugs unnoticed by regular tests then a there are areas which are not covered by regular (functional?) tests and missing tests could be added. If such assumption holds then load/concurrent tests should not less likely to fail if functional tests pass and therefore could be extracted out of Run All. The real world is not ideal so we can use a mentioned idea of significantly limiting load/concurrent tests execution time to 5-10 seconds in Run All. Of course there is a need to make an analysis to find all load/concurrent tests and measure theirs execution time. I think we could use some ML/Data analysis tools for that, could not we? Some additional figures: 1. Number of tests in Run All ~ 50K. 2. Number of test methods in core module ~ 7500. [1] https://github.com/apache/ignite/pull/5419/files пн, 19 нояб. 2018 г. в 14:34, Павлухин Иван : > > Hi, > > I would like to understand following. We are going to make TC green. > We are going to make TC fast. Are we going to do it in parallel? > пн, 19 нояб. 2018 г. в 08:39, Petr Ivanov : > > > > Concerning current TeamCity compute capacity — I think we should invest > > into it’s stability at first: there are lots of problems associated with > > — tests hangs (which now can render agent useless up to 3 days) > > — checkout issues (speedup proxy is installed but sometimes even it is not > > enough for checkout on 100 agents simultaneously, server side checkout > > research is required) > > — tests architecture (current one relies heavily on large file movement > > over network which also can be a bottleneck in case of 100 agents > > simultaneous download start) > > And so on. > > > > After it additional donation can be discussed. > > > > > On 16 Nov 2018, at 17:05, Vyacheslav Daradur wrote: > > > > > > At one of the meetups, Vladimir Ozerov has said that we may specify > > > and move less risky to be broken tests in separate build plan which > > > will be executed daily or weekly > > > > > > For example tests of compatibility with different JDK versions or > > > compatibility between Ignite's releases. > > > > > > Also, I agree with Denis we should find and remove tests with the same > > > checks. > > > > > > By the way, if someone of the community donates TC agents, can this > > > help to reduce the time? > > > > > > On Thu, Nov 15, 2018 at 5:38 PM Yakov Zhdanov wrote: > > >> > > >> Denis, you can go even further. E.g. you can start topology once for the > > >> full set of single threaded full api cache tests. Each test should start > > >> cache dynamically and run it logic. > > >> > > >> As for me, I would think of splitting RunAll to 2 steps - one containing > > >> basic tests and another with more complex tests. 2nd step should not > > >> start > > >> (except manually) if 1st step results in any build failure. > > >> > > >> --Yakov > > > > > > > > > > > > -- > > > Best Regards, Vyacheslav D. > > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: [MTCGA]: new failures in builds [2381922] needs to be handled
Hi Dmitry, You are right - these failures are expected to be muted, I updated Teamcity to account for that. regards, Oleg PS. Readers interested in more details on that matter can find detailed discussion in comments of JIRA ticket IGNITE-10174: https://issues.apache.org/jira/browse/IGNITE-10174?focusedCommentId=16692971=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16692971 "difference in how empty test classes are reported when tests are run..." etc -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
[GitHub] ignite pull request #5485: IGNITE-10354 Failing client node due to not recei...
GitHub user gromtech opened a pull request: https://github.com/apache/ignite/pull/5485 IGNITE-10354 Failing client node due to not receiving metrics updates You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10354 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5485.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5485 commit 5032b5a546c38f9a1d34325bb7a7ea39c66e46e1 Author: Roman Guseinov Date: 2018-11-23T08:26:06Z IGNITE-10354 Failing client node due to not receiving metrics updates ---
Re: [MTCGA]: new failures in builds [2381922] needs to be handled
AFAIK It is expected failure as examples still not fixed. Oleg Ignatenko, could you please mute tests failures? пт, 23 нояб. 2018 г. в 04:55, : > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were > a volunteer to make the contribution to this project, but things change and > you may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and > fix test failures or step down and some committer may revert you commit. > > *Recently contributed test failed in master > TaskExamplesMultiNodeSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-5917517976576215554=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > ClusterGroupExampleSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-878488338790165=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > MonteCarloExamplesMultiNodeSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-924688690363212384=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > ContinuationExamplesMultiNodeSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6047678023796560683=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > TaskExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4133967790505792702=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > DeploymentExamplesMultiNodeSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4750693290259760962=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > MonteCarloExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-1617259290883754314=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > MemcacheRestExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4711279198236915182=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > ContinuationExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-6026321494486034735=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > ContinuousMapperExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=4470984311852304089=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > LifecycleExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4445737465195407541=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > IgfsExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8427106324084470669=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > DeploymentExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7397385867219657348=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > SpringBeanExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3682540866846716810=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > CheckpointExamplesSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7331710564801133366=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > MemcacheRestExamplesMultiNodeSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3180197102757276402=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > ContinuousMapperExamplesMultiNodeSelfTest.initializationError > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-592872012226723313=%3Cdefault%3E=testDetails > Changes may lead to failure were done by > - oignatenko > https://ci.ignite.apache.org/viewModification.html?modId=840076 > - oignatenko > https://ci.ignite.apache.org/viewModification.html?modId=840109 > - aplatonovv > https://ci.ignite.apache.org/viewModification.html?modId=840074 > - dmitrievanthony > https://ci.ignite.apache.org/viewModification.html?modId=839867 > - somefireone > https://ci.ignite.apache.org/viewModification.html?modId=840132 > - kondakov87 >
Re: [MTCGA]: new failures in builds [2371886] needs to be handled
These tests are still failing with issue reference. CacheMvccPartitionedBackupsTest.testBackupsCoherenceWithLargeOperations https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7557959062889759280=%3Cdefault%3E=testDetails CacheMvccReplicatedBackupsTest.testBackupsCoherenceWithLargeOperations https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6825219049988632083=%3Cdefault%3E=testDetails Is there any chance we can get this test fixed/muted? Should I create one more instruction on how to mute test? or can we avoid one more doc hoping JUnit4 will be available soon? Vova, could you step in? чт, 22 нояб. 2018 г. в 11:08, Dmitriy Pavlov : > Hi Igor Seliverstov, > > I can see some test is failed with issue link > https://issues.apache.org/jira/browse/IGNITE-10104 > > Could you please mute test on TC as well? > > Sincerely, > Dmitriy Pavlov > > чт, 22 нояб. 2018 г. в 07:21, : > >> Hi Igniters, >> >> I've detected some new issue on TeamCity to be handled. You are more >> than welcomed to help. >> >> If your changes can lead to this failure(s): We're grateful that you >> were a volunteer to make the contribution to this project, but things >> change and you may no longer be able to finalize your contribution. >> Could you respond to this email and indicate if you wish to continue and >> fix test failures or step down and some committer may revert you commit. >> >> *New test failure in master >> CacheMvccPartitionedBackupsTest.testBackupsCoherenceWithLargeOperations >> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7557959062889759280=%3Cdefault%3E=testDetails >> >> *New test failure in master >> CacheMvccReplicatedBackupsTest.testBackupsCoherenceWithLargeOperations >> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6825219049988632083=%3Cdefault%3E=testDetails >> Changes may lead to failure were done by >> - gvvinblade >> https://ci.ignite.apache.org/viewModification.html?modId=839708 >> - andrey.mashenkov >> https://ci.ignite.apache.org/viewModification.html?modId=839694 >> - pudov.max >> https://ci.ignite.apache.org/viewModification.html?modId=839688 >> - ivandasch >> https://ci.ignite.apache.org/viewModification.html?modId=839748 >> - dmitrievanthony >> https://ci.ignite.apache.org/viewModification.html?modId=839684 >> - jokserfn >> https://ci.ignite.apache.org/viewModification.html?modId=839734 >> >> - Here's a reminder of what contributors were agreed to do >> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute >> - Should you have any questions please contact >> dev@ignite.apache.org >> >> Best Regards, >> Apache Ignite TeamCity Bot >> https://github.com/apache/ignite-teamcity-bot >> Notification generated at 07:21:40 22-11-2018 >> >
[jira] [Created] (IGNITE-10389) Getting affinity for topology version earlier than affinity is calculated
Arnaud Rivero created IGNITE-10389:
--
Summary: Getting affinity for topology version earlier than
affinity is calculated
Key: IGNITE-10389
URL: https://issues.apache.org/jira/browse/IGNITE-10389
Project: Ignite
Issue Type: Bug
Affects Versions: 2.6
Reporter: Arnaud Rivero
When we deploy a new Ignite cluster on YARN and we deploy an Ignite service at
the same time (by configuring it in the xml configuration file et putting it in
the IGNITE_USERS_LIBS path), we get the following exception (the service name
was replaced by XX):
{code:java}
[14:48:42,004][SEVERE][srvc-deploy-#144][GridServiceProcessor] Error when
executing service: XX
java.lang.IllegalStateException: Getting affinity for topology version earlier
than affinity is calculated [locNode=TcpDiscoveryNode
[id=fd770768-189d-4374-82af-2497eff173da, addrs=[127.0.0.1, 127.0.0.3,
172.19.153.13, 172.19.154.13, 172.23.207.13, 192.168.5.1],
sockAddrs=[bt1shujx.bpa.bouyguestelecom.fr/172.23.207.13:47500,
/192.168.5.1:47500, /127.0.0.1:47500,
bt1shujxg2.bpa.bouyguestelecom.fr/172.19.154.13:47500,
bt1shujxg1.bpa.bouyguestelecom.fr/172.19.153.13:47500, /127.0.0.3:47500],
discPort=47500, order=1, intOrder=1, lastExchangeTime=1542894520229, loc=true,
ver=2.6.0#20180710-sha1:669feacc, isClient=false], grp=ignite-sys-cache,
topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0],
head=AffinityTopologyVersion [topVer=4, minorTopVer=1],
history=[AffinityTopologyVersion [topVer=1, minorTopVer=0],
AffinityTopologyVersion [topVer=4, minorTopVer=0], AffinityTopologyVersion
[topVer=4, minorTopVer=1]]]
at
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:632)
at
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:532)
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodesByPartition(GridCacheAffinityManager.java:225)
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryByPartition(GridCacheAffinityManager.java:261)
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryByKey(GridCacheAffinityManager.java:252)
at
org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primaryByKey(GridCacheAffinityManager.java:276)
at
org.apache.ignite.internal.processors.service.GridServiceProcessor$TopologyListener$1.run0(GridServiceProcessor.java:1828)
at
org.apache.ignite.internal.processors.service.GridServiceProcessor$DepRunnable.run(GridServiceProcessor.java:2015)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
We found this previous issue and we wondered if we didn't meet an edge case of
it :
[IGNITE-1171|https://issues.apache.org/jira/browse/IGNITE-1171]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Historical rebalance
Hi Igniters, Currently I’m working on possible approaches how to implement historical rebalance (delta rebalance using WAL iterator) over MVCC caches. The main difficulty is that MVCC writes changes on tx active phase while partition update version, aka update counter, is being applied on tx finish. This means we cannot start iteration over WAL right from the pointer where the update counter updated, but should include updates, which the transaction that updated the counter did. These updates may be much earlier than the point where the update counter was updated, so we have to be able to identify the point where the first update happened. The proposed approach includes: 1) preserve list of active txs, sorted by the time of their first update (using WAL ptr of first WAL record in tx) 2) persist this list on each checkpoint (together with TxLog for example) 4) send whole active tx list (transactions which were in active state at the time the node was crushed, empty list in case of graceful node stop) as a part of partition demand message. 4) find a checkpoint where the earliest tx exists in persisted txs and use saved WAL ptr as a start point or apply current approach in case the active tx list (sent on previous step) is empty 5) start iteration. Your thoughts? Regards, Igor
