[jira] [Created] (IGNITE-13157) Upgrade build process for Javadocs
Mauricio Stekl created IGNITE-13157: --- Summary: Upgrade build process for Javadocs Key: IGNITE-13157 URL: https://issues.apache.org/jira/browse/IGNITE-13157 Project: Ignite Issue Type: Improvement Components: documentation Reporter: Mauricio Stekl Attachments: gsc_mobile_errors.png I am reaching out to you all to see if you could help fix some issues with mobile usability in the Javadocs sections for Ignite website. I know you might think most users will not read the API doc using their smartphones, and that is true. But as you can see in the attached screenshot gsc_mobile_errors.png, we have errors related to mobile usability reported by Google itself through GSC. Obviously this affects our performance on search results, as Google is giving more and more priority to mobile friendly websites. For Apache Ignite website we basically have the largest part of the website with issues, since we only have around 30 pages mobile friendly, and 1000+ pages which are not. Specifically with Javadocs, the main problem is it contains old html frames for layout and navigation, among other markup bad practices, and this would make impossible to update any css to be responsive for small screens. From what I was able to find [[1]|https://bugs.openjdk.java.net/browse/JDK-8215599] [[2]|https://bugs.openjdk.java.net/browse/JDK-8196202], starting with JDK 9 there is a '--no-frames' option which fixes this problem. And with version 11 this option is enabled by default and other features included. You can see here how the new layout looks: [https://docs.oracle.com/en/java/javase/11/docs/api/index.html] Would it be possible to upgrade Java to version 11 only for building the javadocs? This might be a great starting point to fix these problems. Later we could update the .css to adjust font sizes and other details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Webinar: Apache Ignite Management & Monitoring
Hi Everyone, We released a new monitoring and management tool this month for Apache Ignite and GridGain applications; GridGain Control Center. There is a webinar schedule for tomorrow, JUNE 17 2020 10:00AM PDT (-07:00), where I will provide an overview of the features and demo different use cases. If you are interested, the registration link can be found below: https://www.gridgain.com/resources/webinars/simplifying-gridgain-and-apache-ignite-management-with-the-gridgain-control-center If you aren't able to attend, a replay will also be posted shortly after the session. Hope to see you there. Thanks, Greg -- Greg Stachnick Director of Cloud Product Management GridGain Systems (415)407-7192 greg.stachn...@gridgain.com
Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Sergey, Ivan, Could you please check the questions below? If it's time-consuming to rework continuous queries, then the new mode can become available in the experimental state and should not let register continuous queries to avoid potential deadlocks. Overall, this design gap in continuous queries was like a bomb that has just detonated [1]. Anyway, this new connectivity mode will be priceless even if you can't use continuous queries with them because right now we cannot even start a thick client inside of a serverless function. Alexey Plehanov, It looks like we can proceed with the release taking your timelines. [1] https://issues.apache.org/jira/browse/IGNITE-13156 - Denis On Wed, Jun 10, 2020 at 4:16 PM Denis Magda wrote: > Ivan, Sergey, > > How much effort should we put to resolve the issue with > continuous queries? Are you working on that issue actively? Let's try to > guess what would be the ETA. > > - > Denis > > > On Wed, Jun 10, 2020 at 3:55 AM Ivan Bessonov > wrote: > >> Hello, >> >> Sorry for the delay. Sergey Chugunov (sergey.chugu...@gmail.com) just >> replied >> to the main conversation about "communication via discovery" [1]. We work >> on it >> together and recently have found one hard-to-fix scenario, detailed >> description is >> provided in Sergey's reply. >> >> In short, July 10th looks realistic only if we introduce new behavior in >> its current >> implementation, with new setting and IgniteExperimental status. Blocker >> here is >> current implementation of Continuos Query protocol that in some cases >> (described >> at the end) initiates TCP connection right from discovery thread which >> obviously >> leads to deadlock. We haven't estimated efforts needed to redesign of CQ >> protocol >> but it is definitely a risk and fixing it isn't feasible with a code >> freeze at 10th of July. >> So my verdict: we can include this new feature in 2.9 scope as >> experimental and with >> highlighted limitation on CQ usage. Is that OK? >> >> CQ limitation: server needs to open a communication connection to the >> client if during >> CQ registration client tries to p2p deploy new class not available on >> server classpath. >> In other cases registration of CQ should be fine. >> >> [1] >> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-New-Ignite-settings-for-IGNITE-12438-and-IGNITE-13013-td47586.html >> >> вт, 9 июн. 2020 г. в 19:36, Ivan Rakov : >> >>> Hi, >>> >>> Indeed, the tracing feature is almost ready. Discovery, communication and >>> transactions tracing will be introduced, as well as an option to >>> configure >>> tracing in runtime. Right now we are working on final performance >>> optimizations, but it's very likely that we'll complete this activity >>> before the code freeze date. >>> Let's include tracing to the 2.9 release scope. >>> >>> More info: >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-48%3A+Tracing >>> https://issues.apache.org/jira/browse/IGNITE-13060 >>> >>> -- >>> Best Regards, >>> Ivan Rakov >>> >>> On Sat, Jun 6, 2020 at 4:30 PM Denis Magda wrote: >>> >>> > Hi folks, >>> > >>> > The timelines proposed by Alex Plekhanov sounds reasonable to me. I'd >>> like >>> > only to hear inputs of @Ivan Rakov , who is >>> about to >>> > finish with the tracing support, and @Ivan Bessonov >>> > , who is fixing a serious limitation for K8 >>> > deployments [1]. Most likely, both features will be ready by the code >>> > freeze date (July 10), but the guys should know it better. >>> > >>> > [1] >>> > >>> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-New-Ignite-settings-for-IGNITE-12438-and-IGNITE-13013-td47586.html >>> > >>> > - >>> > Denis >>> > >>> > >>> > On Wed, Jun 3, 2020 at 4:45 AM Alex Plehanov >>> > wrote: >>> > >>> >> Hello Igniters, >>> >> >>> >> AI 2.8.1 is finally released and as we discussed here [1] its time to >>> >> start >>> >> the discussion about 2.9 release. >>> >> >>> >> I want to propose myself to be the release manager of the 2.9 release. >>> >> >>> >> What about release time, I agree with Maxim that we should deliver >>> >> features >>> >> as frequently as possible. If some feature doesn't fit into release >>> dates >>> >> we should better include it into the next release and schedule the >>> next >>> >> release earlier then postpone the current release. >>> >> >>> >> I propose the following dates for 2.9 release: >>> >> >>> >> Scope Freeze: June 26, 2020 >>> >> Code Freeze: July 10, 2020 >>> >> Voting Date: July 31, 2020 >>> >> Release Date: August 7, 2019 >>> >> >>> >> WDYT? >>> >> >>> >> [1] : >>> >> >>> >> >>> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Releases-Plan-td47360.html#a47575 >>> >> >>> > >>> >> >> >> -- >> Sincerely yours, >> Ivan Bessonov >> >
Re: [DISCUSSION] New Ignite settings for IGNITE-12438 and IGNITE-13013
Val, I like your suggestion about naming, it describes the purpose of the configuration the best. On Tue, Jun 16, 2020 at 5:18 PM Ivan Bessonov wrote: > Hi, > > I created new issue that describes CQ problem in more details: [1] > I'm fine with experimental status and new property naming. > > [1] https://issues.apache.org/jira/browse/IGNITE-13156 > > вт, 16 июн. 2020 г. в 02:20, Valentin Kulichenko < > valentin.kuliche...@gmail.com>: > > > Folks, > > > > Thanks for providing the detailed clarifications. Let's add the > parameter, > > mark the new feature as experimental, and target for making it the > default > > mode in Ignite 3.0. > > > > I still don't think we can come up with a naming that is really > intuitive, > > but let's try to simplify it as much as possible. How about this: > > > > TcpCommunicationSpi#forceClientToServerConnections -- false by default, > > true if the new mode needs to be enabled. > > > > Let me know your thoughts. > > > > -Val > > > > On Wed, Jun 10, 2020 at 4:10 PM Denis Magda wrote: > > > > > Sergey, > > > > > > Thanks for the detailed explanation and for covering all corner cases. > > > > > > Considering the improvement's criticality, I would continue moving in > the > > > initial direction and add that particular configuration property. > > > Potentially, we can put more effort throughout an Ignite 3.0 timeframe > > and > > > remove the property altogether. @Valentin Kulichenko > > > , could you please suggest any alternate > > naming? > > > > > > Btw, what are the specifics of the issue with continuous queries? It > will > > > be ideal if we could release this new communication option in the GA > > state > > > in 2.9. > > > > > > - > > > Denis > > > > > > > > > On Wed, Jun 10, 2020 at 1:22 AM Sergey Chugunov < > > sergey.chugu...@gmail.com > > > > > > > wrote: > > > > > > > Denis, Val, > > > > > > > > Idea of prohibiting servers to open connections to clients and force > > > > clients to always open "inverse connections" to servers looks > > promising. > > > To > > > > be clear, by "inverse connections" I mean here that server needed to > > > > communicate with client requests client to open a connection back > > instead > > > > of opening connection by itself using addresses published by the > > client. > > > > > > > > If we apply the idea it will indeed allow us to simplify our > > > configuration > > > > (no need for new configuration property), another advantage is > clients > > > > won't need to publish their addresses anymore (with one side note > I'll > > > > cover at the end), it will also simplify our code. > > > > > > > > However applying it with current implementation of inverse connection > > > > request (when request goes across all ring) may bring significant > delay > > > of > > > > opening first connection depending on cluster size and relative > > positions > > > > between server that needs to communicate with client (target server) > > and > > > > client's router node. > > > > > > > > It is possible to overcome this by sending inverse connection request > > not > > > > via discovery but directly to router server node via communication > and > > > > convert to discovery message only on the router. We'll still have two > > > hops > > > > of communication request instead of one plus discovery worker on > client > > > may > > > > be busy working on other stuff slowing down handling of connection > > > request. > > > > But it should be fine. > > > > > > > > However with this solution it is hard to implement failover of router > > > node: > > > > let me describe it in more details. > > > > In case of router node failure target server won't be able to > determine > > > if > > > > client received inverse comm request successfully and (even worse) > > won't > > > be > > > > able to figure out new router for the client without waiting for > > > discovery > > > > event of the client reconnect. > > > > And this brings us to the following choise: we either wait for > > discovery > > > > event about client reconnect (this is deadlock-prone as current > > protocol > > > of > > > > CQ registration opens comm connection to client right from discovery > > > thread > > > > in some cases; if we wait for new discovery event from discovery > > thread, > > > it > > > > is a deadlock) or we fail opening the connection by timeout thus > adding > > > new > > > > scenarios when opening connection may fail. > > > > > > > > Thus implementing communication model "clients connect to servers, > > > servers > > > > never connect to clients" efficiently requires more work on different > > > parts > > > > of our functionality and rigorous testing of readiness of our code > for > > > more > > > > communication connection failures. > > > > > > > > So to me the least risky decision is not to delete new configuration > > but > > > > leave it with experimental status. If we find out that direct request > > > > (server -> router server -> target client) implementation works well > > and > >
Re: [DISCUSSION] New Ignite settings for IGNITE-12438 and IGNITE-13013
Hi, I created new issue that describes CQ problem in more details: [1] I'm fine with experimental status and new property naming. [1] https://issues.apache.org/jira/browse/IGNITE-13156 вт, 16 июн. 2020 г. в 02:20, Valentin Kulichenko < valentin.kuliche...@gmail.com>: > Folks, > > Thanks for providing the detailed clarifications. Let's add the parameter, > mark the new feature as experimental, and target for making it the default > mode in Ignite 3.0. > > I still don't think we can come up with a naming that is really intuitive, > but let's try to simplify it as much as possible. How about this: > > TcpCommunicationSpi#forceClientToServerConnections -- false by default, > true if the new mode needs to be enabled. > > Let me know your thoughts. > > -Val > > On Wed, Jun 10, 2020 at 4:10 PM Denis Magda wrote: > > > Sergey, > > > > Thanks for the detailed explanation and for covering all corner cases. > > > > Considering the improvement's criticality, I would continue moving in the > > initial direction and add that particular configuration property. > > Potentially, we can put more effort throughout an Ignite 3.0 timeframe > and > > remove the property altogether. @Valentin Kulichenko > > , could you please suggest any alternate > naming? > > > > Btw, what are the specifics of the issue with continuous queries? It will > > be ideal if we could release this new communication option in the GA > state > > in 2.9. > > > > - > > Denis > > > > > > On Wed, Jun 10, 2020 at 1:22 AM Sergey Chugunov < > sergey.chugu...@gmail.com > > > > > wrote: > > > > > Denis, Val, > > > > > > Idea of prohibiting servers to open connections to clients and force > > > clients to always open "inverse connections" to servers looks > promising. > > To > > > be clear, by "inverse connections" I mean here that server needed to > > > communicate with client requests client to open a connection back > instead > > > of opening connection by itself using addresses published by the > client. > > > > > > If we apply the idea it will indeed allow us to simplify our > > configuration > > > (no need for new configuration property), another advantage is clients > > > won't need to publish their addresses anymore (with one side note I'll > > > cover at the end), it will also simplify our code. > > > > > > However applying it with current implementation of inverse connection > > > request (when request goes across all ring) may bring significant delay > > of > > > opening first connection depending on cluster size and relative > positions > > > between server that needs to communicate with client (target server) > and > > > client's router node. > > > > > > It is possible to overcome this by sending inverse connection request > not > > > via discovery but directly to router server node via communication and > > > convert to discovery message only on the router. We'll still have two > > hops > > > of communication request instead of one plus discovery worker on client > > may > > > be busy working on other stuff slowing down handling of connection > > request. > > > But it should be fine. > > > > > > However with this solution it is hard to implement failover of router > > node: > > > let me describe it in more details. > > > In case of router node failure target server won't be able to determine > > if > > > client received inverse comm request successfully and (even worse) > won't > > be > > > able to figure out new router for the client without waiting for > > discovery > > > event of the client reconnect. > > > And this brings us to the following choise: we either wait for > discovery > > > event about client reconnect (this is deadlock-prone as current > protocol > > of > > > CQ registration opens comm connection to client right from discovery > > thread > > > in some cases; if we wait for new discovery event from discovery > thread, > > it > > > is a deadlock) or we fail opening the connection by timeout thus adding > > new > > > scenarios when opening connection may fail. > > > > > > Thus implementing communication model "clients connect to servers, > > servers > > > never connect to clients" efficiently requires more work on different > > parts > > > of our functionality and rigorous testing of readiness of our code for > > more > > > communication connection failures. > > > > > > So to me the least risky decision is not to delete new configuration > but > > > leave it with experimental status. If we find out that direct request > > > (server -> router server -> target client) implementation works well > and > > > doesn't bring much complexity in failover scenarios we'll remove that > > > configuration and prohibit servers to open connections to clients by > > > default. > > > > > > Side note: there are rare but yet possible scenarios where client node > > > needs to open communication connection to other client node. If we let > > > clients not to publish their addresses these scenarios will stop > working > > > without additional logic
Re: Save model to JSON
Hello! Our REST Jetty code uses jackson databind. I think it is a safe bet since most users will have it in classpath anyway, since they use our REST. If there are any CVEs we have a lot of pressure to fix it, which will also fix ML export. Regards, -- Ilya Kasnacheev вт, 16 июн. 2020 г. в 15:22, Alexey Zinoviev : > Hi, Igniters! > > I'm going to provide ML model export/import in human-readable format like > JSON. > But I have an issue: what JSON ser/deser library to use: > >- I don't want to add dependency with possible CVEs >- I don't need in special annotations to mark the POJO objects or >something similar >- I need to keep hierarchical (not plain) data in JSON > > Nice to hear any comments/suggestions > > Alexey >
[jira] [Created] (IGNITE-13156) Continuous query filter deployment hungs discovery thread
Ivan Bessonov created IGNITE-13156:
--
Summary: Continuous query filter deployment hungs discovery thread
Key: IGNITE-13156
URL: https://issues.apache.org/jira/browse/IGNITE-13156
Project: Ignite
Issue Type: Bug
Reporter: Ivan Bessonov
Continuous query starts with a custom discovery event. Handler of the event is
executed in discovery thread synchronously. Even worse is the fact that message
itself is mutable and it blocks the ring.
Inside of the handler there is a is p2p resource request from other node, which
can be pretty time consuming. And after
https://issues.apache.org/jira/browse/IGNITE-12438 or similar tasks this could
even lead to a deadlock.
All IO operations must be removed from discovery handlers.
{code:java}
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2099)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2099)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2231)
at
org.apache.ignite.internal.managers.deployment.GridDeploymentCommunication.sendResourceRequest(GridDeploymentCommunication.java:456)
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.sendResourceRequest(GridDeploymentClassLoader.java:793)
at
org.apache.ignite.internal.managers.deployment.GridDeploymentClassLoader.getResourceAsStreamEx(GridDeploymentClassLoader.java:745)
at
org.apache.ignite.internal.managers.deployment.GridDeploymentPerVersionStore.checkLoadRemoteClass(GridDeploymentPerVersionStore.java:729)
at
org.apache.ignite.internal.managers.deployment.GridDeploymentPerVersionStore.getDeployment(GridDeploymentPerVersionStore.java:314)
at
org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getGlobalDeployment(GridDeploymentManager.java:498)
at
org.apache.ignite.internal.GridEventConsumeHandler.p2pUnmarshal(GridEventConsumeHandler.java:416)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1423)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:670)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:533)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2635)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2673)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Save model to JSON
Hi, Igniters! I'm going to provide ML model export/import in human-readable format like JSON. But I have an issue: what JSON ser/deser library to use: - I don't want to add dependency with possible CVEs - I don't need in special annotations to mark the POJO objects or something similar - I need to keep hierarchical (not plain) data in JSON Nice to hear any comments/suggestions Alexey
[jira] [Created] (IGNITE-13155) Snapshot creation throws NPE on an in-memory cluster
Maxim Muzafarov created IGNITE-13155:
Summary: Snapshot creation throws NPE on an in-memory cluster
Key: IGNITE-13155
URL: https://issues.apache.org/jira/browse/IGNITE-13155
Project: Ignite
Issue Type: Bug
Reporter: Maxim Muzafarov
Assignee: Maxim Muzafarov
Fix For: 2.9
Snapshot creation throws NPE on an in-memory cluster.
{code}
Error stack trace:
class org.apache.ignite.internal.client.GridClientException: Failed to handle
request: [req=EXE,
taskName=org.apache.ignite.internal.visor.snapshot.VisorSnapshotCreateTask,
params=[VisorTaskArgument [debug=false]], err=Failed to reduce job results due
to undeclared user exception
[task=org.apache.ignite.internal.visor.snapshot.VisorSnapshotCreateTask@4d45b97f,
err=class org.apache.ignite.IgniteException: null], trace=class
org.apache.ignite.IgniteCheckedException: Failed to reduce job results due to
undeclared user exception
[task=org.apache.ignite.internal.visor.snapshot.VisorSnapshotCreateTask@4d45b97f,
err=class org.apache.ignite.IgniteException: null]
at
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7566)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:260)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:172)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
at
org.apache.ignite.internal.processors.rest.handlers.task.GridTaskCommandHandler$2.apply(GridTaskCommandHandler.java:263)
at
org.apache.ignite.internal.processors.rest.handlers.task.GridTaskCommandHandler$2.apply(GridTaskCommandHandler.java:257)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:399)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:354)
at
org.apache.ignite.internal.processors.rest.handlers.task.GridTaskCommandHandler.handleAsyncUnsafe(GridTaskCommandHandler.java:257)
at
org.apache.ignite.internal.processors.rest.handlers.task.GridTaskCommandHandler.handleAsync(GridTaskCommandHandler.java:163)
at
org.apache.ignite.internal.processors.rest.GridRestProcessor.handleRequest(GridRestProcessor.java:325)
at
org.apache.ignite.internal.processors.rest.GridRestProcessor.access$100(GridRestProcessor.java:104)
at
org.apache.ignite.internal.processors.rest.GridRestProcessor$2.body(GridRestProcessor.java:179)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.compute.ComputeUserUndeclaredException:
Failed to reduce job results due to undeclared user exception
[task=org.apache.ignite.internal.visor.snapshot.VisorSnapshotCreateTask@4d45b97f,
err=class org.apache.ignite.IgniteException: null]
at
org.apache.ignite.internal.processors.task.GridTaskWorker.reduce(GridTaskWorker.java:1184)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:974)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.processDelayedResponses(GridTaskWorker.java:711)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:542)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:830)
at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:555)
at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:535)
at
org.apache.ignite.internal.processors.rest.handlers.task.GridTaskCommandHandler.handleAsyncUnsafe(GridTaskCommandHandler.java:227)
... 8 more
Caused by: class org.apache.ignite.IgniteException: null
at
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1086)
at
org.apache.ignite.internal.util.future.IgniteFutureImpl.convertException(IgniteFutureImpl.java:168)
at
org.apache.ignite.internal.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:137)
at
org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotMXBeanImpl.createSnapshot(SnapshotMXBeanImpl.java:43)
at
org.apache.ignite.internal.visor.snapshot.VisorSnapshotCreateTask$VisorSnapshotCreateJob.run(VisorSnapshotCreateTask.java:57)
at
Re: [DISCUSSION] Ignite integration testing framework.
Hello, Maxim. Thank you for so detailed explanation. Can we put the content of this discussion somewhere on the wiki? So It doesn’t get lost. I divide the answer in several parts. From the requirements to the implementation. So, if we agreed on the requirements we can proceed with the discussion of the implementation. 1. Requirements: The main goal I want to achieve is *reproducibility* of the tests. I’m sick and tired with the zillions of flaky, rarely failed, and almost never failed tests in Ignite codebase. We should start with the simplest scenarios that will be as reliable as steel :) I want to know for sure: - Is this PR makes rebalance quicker or not? - Is this PR makes PME quicker or not? So, your description of the complex test scenario looks as a next step to me. Anyway, It’s cool we already have one. The second goal is to have a strict test lifecycle as we have in JUnit and similar frameworks. > It covers production-like deployment and running a scenarios over a single > database instance. Do you mean «single cluster» or «single host»? 2. Existing tests: > A Combinator suite allows to run set of operations concurrently over given > database instance. > A Consumption suite allows to run a set production-like actions over given > set of Ignite/GridGain versions and compare test metrics across versions > A Yardstick suite > A Stress suite that simulates hardware environment degradation > An Ultimate, DR and Compatibility suites that performs functional regression > testing > Regression Great news that we already have so many choices for testing! Mature test base is a big +1 for Tiden. 3. Comparison: > Criteria: Test configuration > Ducktape: single JSON string for all tests > Tiden: any number of YaML config files, command line option for fine-grained > test configuration, ability to select/modify tests behavior based on Ignite > version. 1. Many YAML files can be hard to maintain. 2. In ducktape, you can set parameters via «—parameters» option. Please, take a look at the doc [1] > Criteria: Cluster control > Tiden: additionally can address cluster as a whole and execute remote > commands in parallel. It seems we implement this ability in the PoC, already. > Criteria: Test assertions > Tiden: simple asserts, also few customized assertion helpers. > Ducktape: simple asserts. Can you, please, be more specific. What helpers do you have in mind? Ducktape has an asserts that waits for logfile messages or some process finish. > Criteria: Test reporting > Ducktape: limited to its own text/HTML format Ducktape have 1. Text reporter 2. Customizable HTML reporter 3. JSON reporter. We can show JSON with the any template or tool. > Criteria: Provisioning and deployment > Ducktape: can provision subset of hosts from cluster for test needs. However, > that means, that test can’t be scaled without test code changes. Does not do > any deploy, relies on external means, e.g. pre-packaged in docker image, as > in PoC. This is not true. 1. We can set explicit test parameters(node number) via parameters. We can increase client count of cluster size without test code changes. 2. We have many choices for the test environment. These choices are tested and used in other projects: * docker * vagrant * private cloud(ssh access) * ec2 Please, take a look at Kafka documentation [2] > I can continue more on this, but it should be enough for now: We need to go deeper! :) [1] https://ducktape-docs.readthedocs.io/en/latest/run_tests.html#options [2] https://github.com/apache/kafka/tree/trunk/tests#ec2-quickstart > 9 июня 2020 г., в 17:25, Max A. Shonichev написал(а): > > Greetings, Nikolay, > > First of all, thank you for you great effort preparing PoC of integration > testing to Ignite community. > > It’s a shame Ignite did not have at least some such tests yet, however, > GridGain, as a major contributor to Apache Ignite had a profound collection > of in-house tools to perform integration and performance testing for years > already and while we slowly consider sharing our expertise with the > community, your initiative makes us drive that process a bit faster, thanks a > lot! > > I reviewed your PoC and want to share a little about what we do on our part, > why and how, hope it would help community take proper course. > > First I’ll do a brief overview of what decisions we made and what we do have > in our private code base, next I’ll describe what we have already donated to > the public and what we plan public next, then I’ll compare both approaches > highlighting deficiencies in order to spur public discussion on the matter. > > It might seem strange to use Python to run Bash to run Java applications > because that introduces IT industry best of breed’ – the Python dependency > hell – to the Java application code base. The only strangest decision one can > made is to use Maven to run Docker to run Bash to run Python to
[jira] [Created] (IGNITE-13154) Introduce aility to manage manage binary types by the users
Taras Ledkov created IGNITE-13154:
-
Summary: Introduce aility to manage manage binary types by the
users
Key: IGNITE-13154
URL: https://issues.apache.org/jira/browse/IGNITE-13154
Project: Ignite
Issue Type: Improvement
Components: binary
Reporter: Taras Ledkov
Assignee: Taras Ledkov
Fix For: 2.9
We need a way to change schema (including unsupported changes, such as column
type change) without cluster restart. This is for the case when all data
associated with the binary type has been removed, so removing the old schema is
safe.
Now users must stop the cluster and remove the folder with binary metadata
manually.
The proposed way is to introduce internal API to manage binary types and public
command line interface (via control.sh). That way one can remove a cache from
the cluster, then unregister corresponding binary types, then launch a new
version of an application that would register the new schema and reload the
data.
*The current implementation has restrictions:*
- all cluster nodes must support remove type feature.
- the cluster must not contains data with type to remove.
- operation of the update type must not be launched on the cluster for the type
to remove (operation examples: put into cache, BinaryObjectBuilder#build).
- client nodes process metadata operation asynchronously so type may be removed
at the client after any delay after the remove type operation is completed.
- if the node that contains the old version of the type joins to the cluster
where type was removed the type is propagated to cluster metadata (because
metadata tombstone not supported).
- if the node that contains the old version of the type cannot join to the
cluster where type was removed and then updated to the new version (because
metadata versioned tombstone not supported).
So, user scenarios looks like:
# Be sure that all server nodes supports remove type feature.
# Remove caches contains the data with type to remove.
# Stops the client node with older version.
# Stops all operation with type to remove (don't create binary objects, don't
run compute jobs with type to remove).
# Remove the type on the stable topology (and production destination topolog).
# Waits any delay (depends on the cluster size and clients count)
# Produce operations with new version of the type.
*Proposed command line interface*
New commands (all commands are _experimental_ ):
- {{--meta list}} prints info about all available binary types:
{{typeId=, typeName=, fields=, schemas=,
isEnum=}}
- {{\-\-meta details (\-\- typeId | \-\-typeName )}} prints detailed
info info about specified type. The type may be specified by type name or type
ID.
output example:
{code}
typeId=0x1FBFBC0C (532659212)
typeName=TypeName1
Fields:
name=fld3, type=long[], fieldId=0x2FFF95 (3145621)
name=fld2, type=double, fieldId=0x2FFF94 (3145620)
name=fld1, type=Date, fieldId=0x2FFF93 (3145619)
name=fld0, type=int, fieldId=0x2FFF92 (3145618)
Schemas:
schemaId=0x6C5CC179 (1818018169), fields=[fld0]
schemaId=0x70E46431 (1894016049), fields=[fld0, fld1, fld2, fld3]
{code}
- {{\-\-meta remove (\-\- typeId | \-\-typeName ) [\-\-out
]}} removes metadata for specified type form cluster and saves the
removed metadata to the specified file. If the file name isn't specified the
output file name is: {{.bin}}
The command requires confirmation.
*N.B.*: The all session of thin clients (ODBC, JDBC, thin client) are closed
(to cleanup local cache of the binary metadata).
- {{\-\-meta update \-\-in ]}} update cluster metadata from
specified file (file name is required)
The command requires confirmation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Reviewer request: [IGNITE-13144] Improve ClusterState enum and other minor improvements for the cluster read-only mode.
Hello, Igniters! I'd like to get a review of the ticket [1]. The patch is ready [2]. Can someone look at it, please? [1] https://issues.apache.org/jira/browse/IGNITE-13144 [2] https://github.com/apache/ignite/pull/7924 -- BR, Sergey Antonov
[jira] [Created] (IGNITE-13153) Fix CammandHandler user Attributes
Sergei Ryzhov created IGNITE-13153: -- Summary: Fix CammandHandler user Attributes Key: IGNITE-13153 URL: https://issues.apache.org/jira/browse/IGNITE-13153 Project: Ignite Issue Type: Task Reporter: Sergei Ryzhov Assignee: Sergei Ryzhov Fix CammandHandler user attributes for used certificate when CammandHandler used with SSL -- This message was sent by Atlassian Jira (v8.3.4#803005)
