Re: Re[2]: [DISCUSS] Page replacement improvement
Hello, Igniters! Zhenya has already approved the patch that adds segmented-LRU and CLOCK page replacement modes [1] (thanks for the review!). Would anyone else like to review it? I have plans to merge it next Monday if nobody else will be interested in reviewing it. [1]: https://issues.apache.org/jira/browse/IGNITE-13761 пт, 25 дек. 2020 г. в 12:26, Alex Plehanov : > Guys, > > I've implemented Segmented-LRU page replacement algorithm and benchmarked > results, it gives some boost (5-10%) when page replacement is > heavily used, but, unfortunately, when replacement is not used it gives > about 2% drop compared to the current Random-LRU page replacement > implementation. Due to this, I think Segmented-LRU can't be used as the > only option or option by default. > > Also, I've implemented CLOCK page replacement algorithm (basic, > not scan-resistant version) and benchmark results are much better. It gives > about the same performance as Segmented-LRU when page replacement is used > and about the same performance as Random-LRU where there is no page > replacement. There are advanced modifications of CLOCK algorithm exists, > but usually, they require additional maintenance cost and we can again get > drop on environments without page replacements compared to Random-LRU. I've > written a benchmark with background full cache scans and even in this case > basic CLOCK version looks promising. > > So, my proposals: > 1. Include all 3 implementations (Random-LRU, Segmented-LRU, CLOCK) to the > product. > 2. Make page replacement algorithm configurable. > 3. Recommend to use Random-LRU for environments with no page replacements > (as it has zero maintenance cost). Recommend to use Segmented-LRU for > environments with a high page replacement rate and a large amount of > one-time scans (as it has near to optimal page to replace selection policy > and scan-resistant). Recommend to use CLOCK for all other cases (as it has > near to zero maintenance cost and efficiency of page replacement near to > Segmented-LRU). > 4. Set CLOCK as the default page replacement algorithm. > > WDYT? > > I've filled the IEP [1] for this discussion and create the pull request > [2] for the last proposal. I would appreciate for review of the patch. > > [1]: > https://cwiki.apache.org/confluence/display/IGNITE/IEP-62+Page+replacement+improvements > [2]: https://github.com/apache/ignite/pull/8513 > > пн, 23 нояб. 2020 г. в 11:12, Zhenya Stanilovsky > : > >> >> >> Nikolay, i hope such case ignite users already observed) >> I suggest to: put data bigger then available, full scan, get data only >> for available inmem data in loop, check PageReplacement metric, how match >> iterations will bring to zero this metric. >> >> >Hello, Alex. >> > >> >> Perhaps we can implement a special benchmark for this case with >> manually triggered "batch page replacement" using yardstick (but I'm not >> sure if ducktape can help here). >> > >> >I think we should carefully describe the issues with the current >> approach and then we can choose right tool to benchmark improvements. >> >You said: >> > >> >> we use Random-LRU algorithm … it has many disadvantages and affects >> performance very much when replacement is started >> > >> >Can you list disadvantages of the Random-LRU? >> > >> >AFAIU the first benchmark should be: >> > >> >1. Start cluster with persistence and put data bigger then available RAM >> to it. >> >2. Measure performance of the queries that selects one entry from the >> cache. >> >3. Make some queries that will iterate throw all data - `SELECT SUM(x) >> FROM t` or something similar. >> >4. Repeat step 2. in this time performance of random queries should be >> lower due to the page replacement. >> > >> >Is this scenario correct? >> > >> >> 23 нояб. 2020 г., в 09:12, Alex Plehanov < plehanov.a...@gmail.com > >> написал(а): >> >> >> >> Nikolay, Zhenya, >> >> >> >> Benchmark from IGNITE-13034 is fully synthetic, it makes random puts >> >> uniformly. It can be used to compare different page replacement >> algorithms >> >> (random-LRU vs segmented-LRU), but it is totally inapplicable to >> benchmark >> >> batch page replacement. >> >> Perhaps we can implement a special benchmark for this case with >> manually >> >> triggered "batch page replacement" using yardstick (but I'm not sure >> >> if ducktape can help here). >> >> >> >>> I understand case you described, but who will pull the switch ? Human, >> >> artificial intelligence ? >> >> As I described before, we can implement both manual and automatic >> "batch >> >> page replacement" triggering. For automatic triggering, there is no >> >> artificial intelligence needed, just several conditions with reasonable >> >> thresholds. Automatic triggering also can be disabled by default. >> >> >> >> пт, 20 нояб. 2020 г. в 13:32, Zhenya Stanilovsky < >> arzamas...@mail.ru.invalid >> >>> : >> >> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> Zhenya, >> >> > Alexey, we already have changes that
Re: Contribute on Apache Ignite.Net
> I wonder if the dev email is a good place to ask questions Yes, dev email is the best place! Don't hesitate to ask anything in this thread or create a new one if needed. On Fri, Jan 22, 2021 at 12:36 AM Nathan Chan wrote: > Hi Pavel, > > Yes, it seems like I missed it, but the weird thing is I have checked my > other email, but I couldn't find the response. However, it is good to hear > from you. Thanks for the suggestion for starting. I am looking at the "How > to Contribute" page now since I want to make sure I understand the SDLC of > Ignite before I start the development. Then I will look at the tickets you > suggested. I wonder if the dev email is a good place to ask questions? or > the Slack channel? > > Please let me know. > > Thanks, > Nathan >
Re: Contribute on Apache Ignite.Net
Hi Pavel, Yes, it seems like I missed it, but the weird thing is I have checked my other email, but I couldn't find the response. However, it is good to hear from you. Thanks for the suggestion for starting. I am looking at the "How to Contribute" page now since I want to make sure I understand the SDLC of Ignite before I start the development. Then I will look at the tickets you suggested. I wonder if the dev email is a good place to ask questions? or the Slack channel? Please let me know. Thanks, Nathan
[MTCGA]: new failures in builds [5835216] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. *New test failure in master JdbcThinTcpIoTest.testExceptionMessage https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1820071520067105235=%3Cdefault%3E=testDetails No changes in the build - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 22:14:48 21-01-2021
Re: [DISCUSSION] Apache Ignite Release 2.10 (time, scope, manager)
Folks, This is the list of bugs which I've cherry-picked to the ignite-2.10 branch: Starvation in mgmt pool caused by MetadataTask execution https://issues.apache.org/jira/browse/IGNITE-13960 .NET: LINQ provider emits incorrect table alias for queries with JOIN and GROUP BY combined https://issues.apache.org/jira/browse/IGNITE-13754 Query cancellation freezes on local lazy queries https://issues.apache.org/jira/browse/IGNITE-13964 SQL: REPLICATED cache cannot be left-joined to PARTITIONED https://issues.apache.org/jira/browse/IGNITE-8732 .NET: logger throws KeyNotFound on creating cache with nullable 'sbyte?' field https://issues.apache.org/jira/browse/IGNITE-12090 Possible deadlock between methods from GridEncryptionManager https://issues.apache.org/jira/browse/IGNITE-13906 OOM on creating rebalance iterator while rebalancing cache with large values. https://issues.apache.org/jira/browse/IGNITE-14003 On Mon, 18 Jan 2021 at 17:20, Pavel Tupitsyn wrote: > > Maxim, > > No objections for .NET-related cherry-picks. > > Thank you! > > On Mon, Jan 18, 2021 at 4:52 PM Никита Сафонов > wrote: > > > Hi Maxim et al, > > > > Thank you very much! > > I do appreciate your help! > > > > Regards, > > Nikita > > > > пн, 18 янв. 2021 г. в 16:16, Maxim Muzafarov : > > > > > Folks, > > > > > > I'd like to cherry-pick these bugs to the ignite-2.10 branch. > > > Do you have any objections? > > > > > > > > > Starvation in mgmt pool caused by MetadataTask execution > > > https://issues.apache.org/jira/browse/IGNITE-13960 > > > > > > .NET: LINQ provider emits incorrect table alias for queries with JOIN > > > and GROUP BY combined > > > https://issues.apache.org/jira/browse/IGNITE-13754 > > > > > > Query cancellation freezes on local lazy queries > > > https://issues.apache.org/jira/browse/IGNITE-13964 > > > > > > SQL: REPLICATED cache cannot be left-joined to PARTITIONED > > > https://issues.apache.org/jira/browse/IGNITE-8732 > > > > > > .NET: logger throws KeyNotFound on creating cache with nullable 'sbyte?' > > > field > > > https://issues.apache.org/jira/browse/IGNITE-12090 > > > > > > Possible deadlock between methods from GridEncryptionManager > > > https://issues.apache.org/jira/browse/IGNITE-13906 > > > > > > On Mon, 18 Jan 2021 at 11:05, Maxim Muzafarov wrote: > > > > > > > > Nikita, > > > > > > > > Thank you for sharing the progress. I will check all the PRs you > > > > reviewed and merge them if need. I will help you with the *[list #3]* > > > > issues (some of them already complete and merged to the 2.10 branch). > > > > > > > > On Fri, 15 Jan 2021 at 22:43, Никита Сафонов < > > vlasovpavel2...@gmail.com> > > > wrote: > > > > > > > > > > Hi Maxim et al, > > > > > > > > > > > > > > > I’d like to share my interim status on Ignite 2.10 documentation > > work. > > > > > > > > > > > > > > > > > > > >- The following PR’s *[list #1]* are reviewed and, when the > > > changes are > > > > >accepted, ready to be merged, including 4 important features. > > > > > > > > > > > > > > > > > > > >- I’m pretty sure that I’ll collect and put together information > > for > > > > >most of the following tasks *[list #2] *by the end of the next > > week, > > > > >including 3 important features. > > > > > > > > > > > > > > > > > > > >- I’d kindly ask your help with the following items *[list #3]*, > > > > >including 2 important features. > > > > > > > > > > > > > > > So, for the sake of documentation completeness, I suggest finalizing > > > our > > > > > doc work next week. > > > > > > > > > > Regards, > > > > > Nikita > > > > > > > > > > > > > > > *[list #1]:* > > > > > > > > > > > > > > > * IGNITE-13968 Add snapshot events documentation pages > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-13968 > > > > > > > > > > https://github.com/apache/ignite/pull/8642 > > > > > > > > > > > > > > > > > > > > * Add documentation for new checkpoint, cluster and cache metrics > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-13982 > > > > > > > > > > > > https://github.com/apache/ignite/pull/8652#pullrequestreview-567256761 > > > > > > > > > > > > > > > > > > > > * Update docs and examples for kubernetes module [IMPORTANT FEATURE] > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-13796?src=confmacro > > > > > > > > > > https://github.com/apache/ignite/pull/8532 > > > > > > > > > > > > > > > > > > > > * TDE - Phase 3. Documentation [IMPORTANT FEATURE] > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-13659?src=confmacro > > > > > > > > > > https://github.com/apache/ignite/pull/8446 > > > > > > > > > > > > > > > > > > > > * Add documentation for the cluster performance profiling tool > > > [IMPORTANT > > > > > FEATURE] > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-13854?src=confmacro > > > > > > > > > > https://github.com/apache/ignite/pull/8663/files > > > > > > > > > > > > > > > > > > > > * Document Persistent store defragmentation
[jira] [Created] (IGNITE-14033) .NET: MessagingTest is flaky
Pavel Tupitsyn created IGNITE-14033: --- Summary: .NET: MessagingTest is flaky Key: IGNITE-14033 URL: https://issues.apache.org/jira/browse/IGNITE-14033 Project: Ignite Issue Type: Bug Components: platforms Reporter: Pavel Tupitsyn Assignee: Pavel Tupitsyn Fix For: 2.11 TestRemoteListen is flaky: https://ci.ignite.apache.org/test/-5844373269997071739?currentProjectId=IgniteTests24Java8=IgniteTests24Java8_PlatformNetCoreLinux=%3Cdefault%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14032) IgniteMessaging.stopRemoteListen does not remove remote listeners immediately
Pavel Tupitsyn created IGNITE-14032:
---
Summary: IgniteMessaging.stopRemoteListen does not remove remote
listeners immediately
Key: IGNITE-14032
URL: https://issues.apache.org/jira/browse/IGNITE-14032
Project: Ignite
Issue Type: Bug
Components: messaging
Reporter: Pavel Tupitsyn
Remote listeners still exist after {{IgniteMessaging.stopRemoteListen}} call,
they are removed after some delay, which is unreliable.
See {{MessagingTest.TestStopRemoteListenRemovesAllCallbacksUponExit}} in
Ignite.NET code as a reproducer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-14031) Add Docker images based on Java 11
Sergey Kadaner created IGNITE-14031: --- Summary: Add Docker images based on Java 11 Key: IGNITE-14031 URL: https://issues.apache.org/jira/browse/IGNITE-14031 Project: Ignite Issue Type: Improvement Reporter: Sergey Kadaner Since Java 11 support was added in https://issues.apache.org/jira/browse/IGNITE-11189 it should be possible to create Docker images as well. This will allow usage of TLSv1.3 and make _‑Djdk.tls.client.protocols=TLSv1.2_ JVM parameter unnecessary on thick clients. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14030) Java thin client: Add containsKeys, clearKey, clearKeys and getAndPutIfAbsent operations to API
Mikhail Petrov created IGNITE-14030: --- Summary: Java thin client: Add containsKeys, clearKey, clearKeys and getAndPutIfAbsent operations to API Key: IGNITE-14030 URL: https://issues.apache.org/jira/browse/IGNITE-14030 Project: Ignite Issue Type: Improvement Reporter: Mikhail Petrov Assignee: Mikhail Petrov It's needed to add containsKeys, clearKey, clearKeys and getAndPutIfAbsent operations to java thin client API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-14029) Inconsistent handling of tables with non primary key columns
Lukas Eder created IGNITE-14029:
---
Summary: Inconsistent handling of tables with non primary key
columns
Key: IGNITE-14029
URL: https://issues.apache.org/jira/browse/IGNITE-14029
Project: Ignite
Issue Type: Bug
Components: sql
Affects Versions: 2.9.1
Reporter: Lukas Eder
This doesn't work:
{code:java}
CREATE TABLE t (i int);{code}
I'm getting this error
{noformat}
SQL Error [1001] [42000]: No PRIMARY KEY defined for CREATE TABLE{noformat}
However, this works perfectly fine:
{code:java}
CREATE TABLE t (i int PRIMARY KEY, dummy int);
ALTER TABLE t DROP dummy;{code}
I must say, I don't fully understand this arbitrary restriction. There are good
reasons for tables to consist only of a primary key, when a schema is properly
normalised. In any case, if there are good reasons to prevent tables without
non primary key columns, then the ALTER TABLE statement must ensure this state
is not achievable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: WAL enable/disable does not work on unstable topology - removal or warning
Ilya, This issue must be fixed for sure (don't think we should rewrite it from scratch). Let's add TODO and warning comment referencing to this issue to the JavaDoc and also add the same warning to documentation pages. The reference to the issue will allow users to track the fixing progress. On Wed, 20 Jan 2021 at 22:39, Mikhail Cherkasov wrote: > > Hi Ilya, > > WAL disable is a very powerful feature that is widely adopted by users. > For sure we need to fix it, even if it means rewriting it. > The warning makes sense, in this case, we can even reduce the priority of > the issue, but anyway, it's at least a critical one, because it can lead to > data loss(and it does). > I would say, instead of a warning, we can do something more noticeable, > like method signature change like: > boolean disableWal(String cacheName, boolean > iReadJavaDockAndAwareOfTheRisk); - this one definitely will be noticed. > > Thanks, > Mike. > > On Wed, Jan 20, 2021 at 8:28 AM Ilya Kasnacheev wrote: > > > Hello! > > > > We had this feature for a few versions, where you could do gnite.cluster(). > > disableWal() to temporarily disable WAL on a specific cache, involving a > > PME and checkpoint on every node. > > > > However, it became apparent that you cannot enable or disable WAL on any > > kind of unstable topology, at all: > > https://issues.apache.org/jira/browse/IGNITE-13976 > > > > You cannot even disable WAL while a baseline node is offline: When it comes > > back, it will not sync its WAL enabled status with the rest of the cluster, > > and all subsequent "WAL enable" or "WAL disable" operations will fail on > > that cache, with no clear way to recover this cache: > > > > ignite.close(); > > client.cluster().disableWal(CACHE_NAME); > > nodes.add(Ignition.start(igniteCfg(false, consistentId))); > > client.cluster().enableWal(CACHE_NAME); // will fail > > > > Even if this simple scenario is fixed, it seems that there are multiple > > failure scenarios if you try to add or remove a node in the middle of WAL > > state change operation. It does not seem that we have any expertise in wal > > disable/enable implementation right now, and I did not find a simple way of > > fixing it short of a full rewrite. > > > > Therefore, I propose that we should *(a) disable that feature* in 2.10 or* > > (b) give a clear warning *when it is used, and also mention in the > > documentation that it may only be used on stable topology. > > > > We may also want to re-mark this feature's API as @IgniteExperimental. > > I have lifted this ticket to be a Blocker. > > > > WDYT? > > > > Regards, > > > > > -- > Thanks, > Mikhail.
[jira] [Created] (IGNITE-14028) Cannot combine COUNT(*) with COUNT(DISTINCT col) in a single query
Lukas Eder created IGNITE-14028:
---
Summary: Cannot combine COUNT(*) with COUNT(DISTINCT col) in a
single query
Key: IGNITE-14028
URL: https://issues.apache.org/jira/browse/IGNITE-14028
Project: Ignite
Issue Type: Bug
Components: sql
Affects Versions: 2.9.1
Reporter: Lukas Eder
Try this query:
{code:java}
CREATE TABLE t (i int PRIMARY KEY, j int);
SELECT count(*), count(DISTINCT i)
FROM t;{code}
I'm getting some sort of internal error, when running the above from DBeaver:
{noformat}
org.jkiss.dbeaver.model.sql.DBSQLException: SQL Error [1] [5]: class
org.apache.ignite.IgniteCheckedException: Index: 0, Size:
0org.jkiss.dbeaver.model.sql.DBSQLException: SQL Error [1] [5]: class
org.apache.ignite.IgniteCheckedException: Index: 0, Size: 0 at
org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:133)
at
org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeStatement(SQLQueryJob.java:509)
at
org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.lambda$0(SQLQueryJob.java:440)
at
org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:168)
at
org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeSingleQuery(SQLQueryJob.java:427)
at
org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.extractData(SQLQueryJob.java:812)
at
org.jkiss.dbeaver.ui.editors.sql.SQLEditor$QueryResultsContainer.readData(SQLEditor.java:3180)
at
org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.lambda$0(ResultSetJobDataRead.java:121)
at
org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:168)
at
org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.run(ResultSetJobDataRead.java:119)
at
org.jkiss.dbeaver.ui.controls.resultset.ResultSetViewer$ResultSetDataPumpJob.run(ResultSetViewer.java:4512)
at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:105) at
org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)Caused by:
java.sql.SQLException: class org.apache.ignite.IgniteCheckedException: Index:
0, Size: 0 at
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:1004)
at
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:231)
at
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:559)
at
org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.execute(JDBCStatementImpl.java:327)
at
org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:130)
... 12 more{noformat}
A workaround:
{code:java}
SELECT count(1), count(DISTINCT i)
FROM t;{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-14027) Switching Auto-adjust on does not trigger including existent nodes(which are not in BLT) to the BLT
Semyon Danilov created IGNITE-14027: --- Summary: Switching Auto-adjust on does not trigger including existent nodes(which are not in BLT) to the BLT Key: IGNITE-14027 URL: https://issues.apache.org/jira/browse/IGNITE-14027 Project: Ignite Issue Type: Bug Components: general Affects Versions: 2.9.1 Reporter: Semyon Danilov Assignee: Semyon Danilov Steps to reproduce: * Start cluster with AA on * Disable AA * Start new node - it joins topology, not the BL * Enable AA Expected: * As _softTimeout_ passes, the node joins the BLT Actual: * The node remains out of BLT With that new note won't join BLT until either another new node started or AA disabled and the node is included in the BLT manually -- This message was sent by Atlassian Jira (v8.3.4#803005)
