[jira] [Created] (IGNITE-13277) ignite-zookeeper does not work with Java 14
Max Shonichev created IGNITE-13277:
--
Summary: ignite-zookeeper does not work with Java 14
Key: IGNITE-13277
URL: https://issues.apache.org/jira/browse/IGNITE-13277
Project: Ignite
Issue Type: Bug
Reporter: Max Shonichev
ignite-zookeeper module depends on 3.4 zookeeper branch, which is known to fail
with Java 14, see https://issues.apache.org/jira/browse/ZOOKEEPER-3779
Symptom:
{noformat}
Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=3
watcher=org.apache.ignite.spi.discovery.zk.internal.ZookeeperClient@49665f92
[2020-07-20 18:18:43,512][WARN
][zk-null-SendThread(127.0.0.1:2181)][ClientCnxn] Session 0x0 for server
127.0.0.1/:2181, unexpected error, closing socket connection and
attempting reconnect
java.lang.IllegalArgumentException: Unable to canonicalize address
127.0.0.1/:2181 because it's not resolvable
{noformat}
Workaround: none yet, need to upgrade ignite-zookeeper to 3.5 branch or wait
ticket ZOOKEEPER-3779 closed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Getting rid of NONE cache rebalance mode
+1 for deprecating/removing NONE mode. Alexey, what do you think about the SYNC mode? In my experience, it does not add much value as well. I would go as far as removing the rebalancingMode parameter altogether (probably in 3.0). -Val On Mon, Jul 20, 2020 at 11:09 AM Ivan Pavlukhin wrote: > Alexey, Igniters, > > Could you please outline motivation answering following questions? > 1. Does this mode generally work correctly today? > 2. Can this mode be useful at all? > > I can imagine that it might be useful in a transparent caching use > case (if I did not misunderstand). > > 2020-07-20 20:39 GMT+03:00, Pavel Tupitsyn : > > +1 > > > > More evidence: > > > https://stackoverflow.com/questions/62902640/apache-ignite-cacherebalancemode-is-not-respected-by-nodes > > > > On Mon, Jul 20, 2020 at 8:26 PM Alexey Goncharuk > > > > wrote: > > > >> Igniters, > >> > >> I would like to run the idea of deprecating and probably ignoring the > >> NONE > >> rebalance mode by the community. It's in the removal list for Ignite 3.0 > >> [1], but it looks like it still confuses and creates issues for users > >> [2]. > >> > >> What about deprecating it in one of the next releases and even ignoring > >> this constant in further releases, interpreting it as ASYNC, before > >> Ignite > >> 3.0? I find it hard to believe that any Ignite user actually has > >> RebalanceMode.NONE set in their configuration due to its absolutely > >> unpredictable behavior. > >> > >> Thanks for your thoughts, > >> --AG > >> > >> [1] > >> > >> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist > >> [2] > >> > >> > http://apache-ignite-developers.2346864.n4.nabble.com/About-Rebalance-Mode-SYNC-amp-NONE-td47279.html > >> > > > > > -- > > Best regards, > Ivan Pavlukhin >
Re: Getting rid of NONE cache rebalance mode
Alexey, Igniters, Could you please outline motivation answering following questions? 1. Does this mode generally work correctly today? 2. Can this mode be useful at all? I can imagine that it might be useful in a transparent caching use case (if I did not misunderstand). 2020-07-20 20:39 GMT+03:00, Pavel Tupitsyn : > +1 > > More evidence: > https://stackoverflow.com/questions/62902640/apache-ignite-cacherebalancemode-is-not-respected-by-nodes > > On Mon, Jul 20, 2020 at 8:26 PM Alexey Goncharuk > > wrote: > >> Igniters, >> >> I would like to run the idea of deprecating and probably ignoring the >> NONE >> rebalance mode by the community. It's in the removal list for Ignite 3.0 >> [1], but it looks like it still confuses and creates issues for users >> [2]. >> >> What about deprecating it in one of the next releases and even ignoring >> this constant in further releases, interpreting it as ASYNC, before >> Ignite >> 3.0? I find it hard to believe that any Ignite user actually has >> RebalanceMode.NONE set in their configuration due to its absolutely >> unpredictable behavior. >> >> Thanks for your thoughts, >> --AG >> >> [1] >> >> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist >> [2] >> >> http://apache-ignite-developers.2346864.n4.nabble.com/About-Rebalance-Mode-SYNC-amp-NONE-td47279.html >> > -- Best regards, Ivan Pavlukhin
Re: Getting rid of NONE cache rebalance mode
+1 More evidence: https://stackoverflow.com/questions/62902640/apache-ignite-cacherebalancemode-is-not-respected-by-nodes On Mon, Jul 20, 2020 at 8:26 PM Alexey Goncharuk wrote: > Igniters, > > I would like to run the idea of deprecating and probably ignoring the NONE > rebalance mode by the community. It's in the removal list for Ignite 3.0 > [1], but it looks like it still confuses and creates issues for users [2]. > > What about deprecating it in one of the next releases and even ignoring > this constant in further releases, interpreting it as ASYNC, before Ignite > 3.0? I find it hard to believe that any Ignite user actually has > RebalanceMode.NONE set in their configuration due to its absolutely > unpredictable behavior. > > Thanks for your thoughts, > --AG > > [1] > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist > [2] > > http://apache-ignite-developers.2346864.n4.nabble.com/About-Rebalance-Mode-SYNC-amp-NONE-td47279.html >
Getting rid of NONE cache rebalance mode
Igniters, I would like to run the idea of deprecating and probably ignoring the NONE rebalance mode by the community. It's in the removal list for Ignite 3.0 [1], but it looks like it still confuses and creates issues for users [2]. What about deprecating it in one of the next releases and even ignoring this constant in further releases, interpreting it as ASYNC, before Ignite 3.0? I find it hard to believe that any Ignite user actually has RebalanceMode.NONE set in their configuration due to its absolutely unpredictable behavior. Thanks for your thoughts, --AG [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist [2] http://apache-ignite-developers.2346864.n4.nabble.com/About-Rebalance-Mode-SYNC-amp-NONE-td47279.html
[jira] [Created] (IGNITE-13276) .NET: Enable multi-process tests in DotNetCore project
Pavel Tupitsyn created IGNITE-13276: --- Summary: .NET: Enable multi-process tests in DotNetCore project Key: IGNITE-13276 URL: https://issues.apache.org/jira/browse/IGNITE-13276 Project: Ignite Issue Type: Improvement Components: platforms Reporter: Pavel Tupitsyn `IgniteProcess` class is not cross-platform, so related tests don't work on .NET Core and in non-Windows environments. Make it cross-platform and enable related tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13275) .NET: Add peer assembly loading tests to .NET Core project
Pavel Tupitsyn created IGNITE-13275: --- Summary: .NET: Add peer assembly loading tests to .NET Core project Key: IGNITE-13275 URL: https://issues.apache.org/jira/browse/IGNITE-13275 Project: Ignite Issue Type: Improvement Components: platforms Reporter: Pavel Tupitsyn Peer assembly loading tests are excluded from Apache.Ignite.Core.Tests.DotNetCore currently due to Examples dll usage. Fix this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13274) .NET: IgniteConfiguration.ClusterStateOnStart
Pavel Tupitsyn created IGNITE-13274: --- Summary: .NET: IgniteConfiguration.ClusterStateOnStart Key: IGNITE-13274 URL: https://issues.apache.org/jira/browse/IGNITE-13274 Project: Ignite Issue Type: Improvement Components: platforms Reporter: Pavel Tupitsyn Fix For: 2.10 Add IgniteConfiguration.ClusterStateOnStart, mark IgniteConfiguration.IsActiveOnStart as obsolete -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13273) .NET: SqlConfiguration
Pavel Tupitsyn created IGNITE-13273: --- Summary: .NET: SqlConfiguration Key: IGNITE-13273 URL: https://issues.apache.org/jira/browse/IGNITE-13273 Project: Ignite Issue Type: Improvement Components: platforms Reporter: Pavel Tupitsyn Fix For: 2.10 Add IgniteConfiguration.SqlConfiguration -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13272) .NET: ShutdownPolicy
Pavel Tupitsyn created IGNITE-13272: --- Summary: .NET: ShutdownPolicy Key: IGNITE-13272 URL: https://issues.apache.org/jira/browse/IGNITE-13272 Project: Ignite Issue Type: Improvement Components: platforms Reporter: Pavel Tupitsyn Fix For: 2.10 Add IgniteConfiguration.ShutdownPolicy and ICluster.ShutdownPolicy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Welcome message
Hi Aleksandr, Welcome to Apache Ignite Community! I added contributor permissions to your account. Happy contributing! 2020-07-20 16:31 GMT+03:00, Alex Kozhenkov : > Hi, all! > > I'm Aleksandr Kozhenkov, software engineer from Russia. > I want to contribute to Apache Ignite. Please, give me access to Jira (my > username is "wirtzleg") > Thank you. > -- Best regards, Ivan Pavlukhin
Welcome message
Hi, all! I'm Aleksandr Kozhenkov, software engineer from Russia. I want to contribute to Apache Ignite. Please, give me access to Jira (my username is "wirtzleg") Thank you.
[jira] [Created] (IGNITE-13271) Add new type of WAL records to track/debug atomic updates on backup nodes
Vyacheslav Koptilin created IGNITE-13271: Summary: Add new type of WAL records to track/debug atomic updates on backup nodes Key: IGNITE-13271 URL: https://issues.apache.org/jira/browse/IGNITE-13271 Project: Ignite Issue Type: Task Reporter: Vyacheslav Koptilin Assignee: Vyacheslav Koptilin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13270) Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5
Tanmay Ambre created IGNITE-13270: - Summary: Massive CPU burn in 2.8.0 and 2.8.1 after upgrading from 2.7.5 Key: IGNITE-13270 URL: https://issues.apache.org/jira/browse/IGNITE-13270 Project: Ignite Issue Type: Bug Components: sql Affects Versions: 2.8.1, 2.8 Reporter: Tanmay Ambre After upgrading to ignite 2.8.0 from 2.7.5 the following query is causing massive CPU burn. We have a 8 node server cluster for Ignite (with 32 CPUs on each node). with 2.8.0, 2 nodes are maxed out on CPU. with 2.8.1 all nodes are above 80%. On 2.7.5 the same query was performing very nicely. The query is a join with 3 tables having same affinity key. however, we dont know the affinity key when we are querying and there this results in a merge scan. Query: select A.A_ID, A.AFFINITYKEY, B.B_ID from A, B, C where A.A_ID = B.A_ID AND B.B_ID = C.B_ID AND A.AFFINITYKEY = B.AFFINITYKEY AND B.AFFINITYKEY = C.AFFINITYKEY AND B.B_ID = ? AND C.C_ID = ? Performance: 2.7.5 ~ 2 to 5 ms 2.8.0 ~ 6 to 12 seconds 2.8.1 ~ within 3 seconds Volume (we have a EDA based system where messages come in burst. One burst has approx. 150K events - each event correspond to one query). throughput of our java based processor (that fires this query): 2.7.5 ~ 750 transactions per second 2.8.0 ~ 3 transactions per second 2.8.1 ~ 6 transactions per second CPU burn 2.7.5 ~ 10 to 15% on each node 2.8.0 ~ 100% on 2 nodes other nodes are idling 2.8.1 ~ 80 to 90% on each node -- This message was sent by Atlassian Jira (v8.3.4#803005)
TcpCommunicationSpi split to independent classes [IGNITE-13098]
Hi, I'm almost done donating these changes from the GridGain repository. It has been tested and published in several releases. I would like to encourage you to make the same changes because the product testing process is very difficult. These changes usually do not affect the behavior and public API. In the next patch, I will pass a Dependencyresolver that avoids using reflection in tests. Issues - https://issues.apache.org/jira/browse/IGNITE-13098 Draft here - https://github.com/apache/ignite/pull/8057 Description This ticket describes requirements for TcpCommunicationSpi refactoring. The main goal is to split the class without changing behavior and public API. *Actual problem:* CurrentlyTcpCommunicationSpi has over 5K lines and includes about15+ inner classes like: 1. ShmemAcceptWorker 2. SHMemHandshakeClosure 3. ShmemWorker 4. CommunicationDiscoveryEventListener 5. CommunicationWorker 6. ConnectFuture 7. ConnectGateway 8. ConnectionKey 9. ConnectionPolicy 10. DisconnectedSessionInfo 11. FirstConnectionPolicy 12. HandshakeTimeoutObject 13. RoundRobinConnectionPolicy 14. TcpCommunicationConnectionCheckFuture 15. TcpCommunicationSpiMBeanImpl In addition, it contains logic of client connection life cycle, nio server handler, and handshake handler. The classes above have cyclic dependencies and high coupling.The whole mechanism works because classes have access to each other via parent class references. As a result, initialization of class isn't consistent. By consistent I mean that class created via constructor is ready to be used. All of the classes work with context and shareproperties everywhere. Many methods of TcpCommunicationSpi don’t have a single responsibility. Example is getNodeAttribute:,it makes client reservation, takes the IP address of the node and provides attributes. It works fine and we usually don’t have reasons to change anything. But if you want to create a test that has a little different behavior than a blocking message, you can't mock or change the behavior of inner classes. For example, test covering change in the handshake process. Some people make test methods in public API like "closeConnections" or "openSocketChannel" because the current design isn't fine for it. It also takes a lot of time for test development for minor changes. *Solution:* The scope of work is big and communication spi is place which should be changed carefully. I recommend to make this refactoring step by step. - The first idea is to split the parent class into independent classes and move them to the internal package. We should achieveSOLID when it’s done. - Extract spread logic to appropriate classes like ClientPool, HandshakeHandler, etc. - Make a common transfer object for TCSpi configuration. - Make dependencies direct if it is possible. - Initialize all dependencies in one place. - Make child classes context-free. - Try to do classes more testable. - Use the idea of dependency injection without a framework for it. *Benefits:* With the ability to write truly jUnit-style tests and cover functionality with better testing we get a way to easier develop new features and optimizations needed in such low-level components as TcpCommunicationSpi. Examples of features that improve usability of Apache Ignite a lot are: inverse communication connection with optimizations and connection multiplexing. Both of the features could be used in environments with restricted network connectivity (e.g. when connections between nodes could be established only in one direction).
Re: [DISCUSSION] Add index rebuild time metrics
Stan, Currently we never build indexes one-by-one - we always use a cache data row visitor which either updates all indexes (see IndexRebuildFullClosure) or updates a set of all indexes that need to catch up (see IndexRebuildPartialClosure). GIven that, I do not see any need for per-index rebuild status as this status will be updated for all outdated indexes simultaneously. Kirill's approach for the total number of processed keys per cache seems reasonable to me. --AG пт, 3 июл. 2020 г. в 10:12, ткаленко кирилл : > Hi, Stan! > > Perhaps it is worth clarifying what exactly I wanted to say. > Now we have 2 processes: building and rebuilding indexes. > > At moment, we have some metrics for rebuilding indexes: > "IsIndexRebuildInProgress", "IndexBuildCountPartitionsLeft". > > I suggest adding another metric "Indexrebuildkeyprocessed", which will > allow you to determine how many records are left to rebuild for cache. > > I think your comments are more about building an index that may need more > metrics, but I think you should do it in a separate ticket. > > 03.07.2020, 03:09, "Stanislav Lukyanov" : > > If multiple indexes are to be built "number of indexed keys" metric may > be misleading. > > > > As a cluster admin, I'd like to know: > > - Are all indexes ready on a node? > > - How many indexes are to be built? > > - How much resources are used by the index building (how many threads > are used)? > > - Which index(es?) is being built right now? > > - How much time until the current (single) index building finishes? Here > "time" can be a lot of things: partitions, entries, percent of the cache, > minutes and hours > > - How much time until all indexes are built? > > - How much does it take to build each of my indexes / a single index of > my cache on average? > > > > I think we need a set of metrics and/or log messages to solve all of > these questions. > > I imaging something like: > > - numberOfIndexesToBuild > > - a standard set of metrics on the index building thread pool (do we > already have it?) > > - currentlyBuiltIndexName (assuming we only build one at a time which is > probably not true) > > - for the "time" metrics I think percentage might be the best as it's > the easiest to understand; we may add multiple metrics though. > > - For "time per each index" I'd add detailed log messages stating how > long did it take to build a particular index > > > > Thanks, > > Stan > > > >> On 26 Jun 2020, at 12:49, ткаленко кирилл > wrote: > >> > >> Hi, Igniters. > >> > >> I would like to know if it is possible to estimate how much the index > rebuild will take? > >> > >> At the moment, I have found the following metrics [1] and [2] and > since the rebuild is based on caches, I think it would be useful to know > how many records are processed in indexing. This way we can estimate how > long we have to wait for the index to be rebuilt by subtracting [3] and how > many records are indexed. > >> > >> I think we should add this metric [4]. > >> > >> Comments, suggestions? > >> > >> [1] - https://issues.apache.org/jira/browse/IGNITE-12184 > >> [2] - > org.apache.ignite.internal.processors.cache.CacheGroupMetricsImpl#idxBuildCntPartitionsLeft > >> [3] - org.apache.ignite.cache.CacheMetrics#getCacheSize > >> [4] - org.apache.ignite.cache.CacheMetrics#getNumberIndexedKeys >
Re: Moving Ignite documentation to github
Denis, How about the next step of taking the HTML and committing it to the website repository? Did you have a chance to think through this step? Yes, I'll look into this this week. This shouldn't be very difficult. -Artem On 18.07.2020 00:43, Denis Magda wrote: Worked out well on my end. Thanks for sending the update! How about the next step of taking the HTML and committing it to the website repository? Did you have a chance to think through this step? - Denis On Fri, Jul 17, 2020 at 5:27 AM Artem Budnikov wrote: Hi everyone, I've prepared the initial set of source files for the Ignite documentation. If you are interested, you can take a look at https://github.com/apache/ignite/tree/IGNITE-7595/docs You can run a local web-server (jekyll) if you want to view the docs in your browser. Refer to the README.adoc for instructions. Some people had troubles installing Jekyll locally, so I added an instruction on how to use jekyll docker image. If you have any comments on the overall approach, please let me know. The styles and content are still a work in progress, so please don't report issues related to that. -Artem On 26.06.2020 01:54, Guru Stron wrote: +1 for migrating docs to github. It will allow an easier contribution for docs, I think. As a nice feature - adding an edit link (submit PR for docs) to the document page on site. As for keeping them separate - Microsoft keeps docs for it's products in separate repos, for example. On Thu, 25 Jun 2020 at 15:48, Artem Budnikov < a.budnikov.ign...@gmail.com> wrote: OK, let's give it a try. The way I see it, the documentation source files will be located in the "/docs" folder, including UI templates/styles, asciidoc files, and build scripts. I'll start experimenting with this and will let you know when basic setup is ready. -Artem On 23.06.2020 20:19, Denis Magda wrote: I believe that by keeping the documentation sources in the same repository with the source code will help us to prepare and release all the release artifacts at the same time. So, +1 for hosting raw documentation ascii-doc pages in the main Ignite repo. However, the HTML version needs to reside on the Ignite website, which is similar to the API docs. We can create tools to do this in one click. Post-reviews are not prohibited in Apache, quite the opposite, and they suit the documentation contribution process better. It's ok if committers to the documentation merge the changes first and ask for a review later if needed. - Denis On Tue, Jun 23, 2020 at 6:53 AM Artem Budnikov < a.budnikov.ign...@gmail.com> wrote: Pavel, I don't think so: we can't add snippets pointing to new APIs from a separate repo, Snippets are kept together with the docs, they /don't need/ to be stored in the main repo, although they can. They are compilable and up to date. I update the docs and API samples for features that hasn't been released in the GridGain docs and never thought it was a problem. I understand that you don't want to do extra work when adding code samples, but it looks like just an inconvenience. Let me suggest this: Let's think about a solution that will be comfortable for you, I'm pretty sure this inconvenience can be solved technically. But I need time to think it through. we can't see the docs when doing global search (and/or replace) from the IDE. I think you can add the docs repo to your IDE as a project. I used to do it in the beginning but then switched to Sublime Text, because it's more convenient to me. We are looking at it from different perspectives. I'm trying to create a process that is comfortable for tech writers rather than developers. And everybody has to accept some kind of a compromise:) Well, no one is able to "freely" commit code to Apache master, there is a process to follow - CI, reviews, etc. Same should happen for the docs, separate repo or not. But a separate repo will require separate ownership/management (probably?), but we already have everything in the main repo, why introduce overhead? Just think about it from my perspective. That creates a HUUUGE overhead for technical writers who work on the docs, and they are the ones who provide 90% of updates. I agree about the review process, and I'm going to think it over. But now it seems that we don't have to impose any strict process that impedes preparation of the docs. -Artem On 23.06.2020 15:35, Pavel Tupitsyn wrote: all your pros points work just as well for a separate repository I don't think so: we can't add snippets pointing to new APIs from a separate repo, we can't see the docs when doing global search (and/or replace) from the IDE. I am able to freely commit to master Well, no one is able to "freely" commit code to Apache master, there is a process to follow - CI, reviews, etc. Same should happen for the docs, separate repo or not. But a separate repo will require separate ownership/management (probably?), but we already have everything
Re: Proposal of new event QUERY_EXECUTION_EVENT
Looks like EVT_CACHE_QUERY_EXECUTED just works for different use cases: 1. it relates to a specific cache (Event for SQL queries looks different as it could contain multiple caches or none of them); 2. Also the EVT_CACHE_QUERY_EXECUTED event fires multiple times for distributed queries, see GridMapQueryExecutor class (For SQL query it's required to fire once independently on how many nodes are affected). So there are different patterns for events. I think EVT_CACHE_QUERY_EXECUTED should not be deprecated or changed. > What happens in case the event listener fails? > Would the event notification be synchronous? It depends on how other events are implemented. As I see for the EVT_CACHE_QUERY_EXECUTED event - it's synchronous, and listener errors aren't handled. I think these questions are related to GridEventStorageManager as it just provides an API for recording events. Internal implementations (sync / async, error handling) is not related to an event, is it? > I have some doubts about provide text of a query even with hidden arguments, probably it should be configured due to it could lead to security leak Currently event EVT_CACHE_QUERY_EXECUTED provides a sql clause without limitations. If we're going to provide some restrictions it will require additional investigation. I see at least 2 configurations here: 1. Ignite can be configured to hide clase, params only or nothing for all listeners; 2. Only authorized listeners can subscribe to the event. Should we discuss this within this topic? On Mon, Jul 20, 2020 at 1:55 PM Юрий wrote: > In my opinion existing events EVT_CACHE_QUERY_EXECUTION_STARTED should be > deprecated and added two new for start and for end of queries which should > cover all SQL query types. > I have some doubts about provide text of a query even with hidden > arguments, probably it should be configured due to it could lead to > security leak > > пн, 20 июл. 2020 г. в 12:49, Stanislav Lukyanov : > > > Maksim, > > > > Can we change the EVT_CACHE_QUERY_EXECUTED to fire earlier? Or should > > there be an EVT_CACHE_QUERY_EXECUTION_STARTED for the query start, while > > the old event would continue to work for query finish? > > I think the new event needs to either reuse the old one, or be its mirror > > for the query start. It also means that we probably should resolve the > > issues you've listed. > > > > Would the event notification be synchronous? In which thread? > Asynchronous > > is generally preferred - would it work? > > > > What happens in case the event listener fails? > > > > Thanks, > > Stan > > > > > On 16 Jul 2020, at 18:49, Denis Magda wrote: > > > > > > Taras, Yury, Ivan, > > > > > > Could you please join this thread and share your thoughts? Do we > already > > > have any plans on tracking of the DDL and DML queries? > > > > > > - > > > Denis > > > > > > > > > On Wed, Jul 15, 2020 at 12:09 AM Max Timonin > > > wrote: > > > > > >> Hi Denis, thanks for the answer! > > >> > > >> We already checked EVT_CACHE_QUERY_EXECUTED and found that it works > > only in > > >> cases: > > >> 1. Scan queries and Select queries (common pattern is access to cache > > >> data); > > >> 2. This event triggers only if query execution succeeds, in case of > > failure > > >> while execution this event won't fire. > > >> > > >> Our additional requirements are to protocol queries: > > >> 1. that aren't cache related (for example, alter user); > > >> 2. that relate to multiple caches (while EVT_CACHE_QUERY_EXECUTED have > > >> field cacheName related to specific cache); > > >> 3. we need to protocol also DDL and DML queries. > > >> > > >> Regards, > > >> Maksim > > >> > > >> On Tue, Jul 14, 2020 at 10:20 PM Denis Magda > wrote: > > >> > > >>> Hi Max, > > >>> > > >>> Could you check if the EVT_CACHE_QUERY_EXECUTED event is what you're > > >>> looking for? > > >>> > > >>> > > >> > > > https://www.gridgain.com/docs/latest/developers-guide/events/events#cache-query-events > > >>> > > >>> - > > >>> Denis > > >>> > > >>> > > >>> On Fri, Jul 10, 2020 at 3:54 AM Max Timonin > > > >>> wrote: > > >>> > > Hi Igniters! > > > > We're going to protocol all input SQL queries from our users. > > Currently > > there is no such mechanism in Ignite to use for it. So we're > proposing > > >> to > > add a new event: QUERY_EXECUITION_EVENT. > > > > Requirements for the event: > > 1. If this event fires it means that a query is correct and will be > > executed (and failed only in exceptional cases); > > > > 2. Event fires for all query types; > > > > 3. Required fields are: > > - text of a query (with hidden arguments); > > - arguments of query; > > - query type; > > - node id. > > > > Looks that this event should go along with `runningQryMgr::register` > > in > > class `IgniteH2Indexing` as this method invoked for all input > queries > > >>> too. > > > > What do you think? > > > > Regards, > >
Re: Proposal of new event QUERY_EXECUTION_EVENT
In my opinion existing events EVT_CACHE_QUERY_EXECUTION_STARTED should be deprecated and added two new for start and for end of queries which should cover all SQL query types. I have some doubts about provide text of a query even with hidden arguments, probably it should be configured due to it could lead to security leak пн, 20 июл. 2020 г. в 12:49, Stanislav Lukyanov : > Maksim, > > Can we change the EVT_CACHE_QUERY_EXECUTED to fire earlier? Or should > there be an EVT_CACHE_QUERY_EXECUTION_STARTED for the query start, while > the old event would continue to work for query finish? > I think the new event needs to either reuse the old one, or be its mirror > for the query start. It also means that we probably should resolve the > issues you've listed. > > Would the event notification be synchronous? In which thread? Asynchronous > is generally preferred - would it work? > > What happens in case the event listener fails? > > Thanks, > Stan > > > On 16 Jul 2020, at 18:49, Denis Magda wrote: > > > > Taras, Yury, Ivan, > > > > Could you please join this thread and share your thoughts? Do we already > > have any plans on tracking of the DDL and DML queries? > > > > - > > Denis > > > > > > On Wed, Jul 15, 2020 at 12:09 AM Max Timonin > > wrote: > > > >> Hi Denis, thanks for the answer! > >> > >> We already checked EVT_CACHE_QUERY_EXECUTED and found that it works > only in > >> cases: > >> 1. Scan queries and Select queries (common pattern is access to cache > >> data); > >> 2. This event triggers only if query execution succeeds, in case of > failure > >> while execution this event won't fire. > >> > >> Our additional requirements are to protocol queries: > >> 1. that aren't cache related (for example, alter user); > >> 2. that relate to multiple caches (while EVT_CACHE_QUERY_EXECUTED have > >> field cacheName related to specific cache); > >> 3. we need to protocol also DDL and DML queries. > >> > >> Regards, > >> Maksim > >> > >> On Tue, Jul 14, 2020 at 10:20 PM Denis Magda wrote: > >> > >>> Hi Max, > >>> > >>> Could you check if the EVT_CACHE_QUERY_EXECUTED event is what you're > >>> looking for? > >>> > >>> > >> > https://www.gridgain.com/docs/latest/developers-guide/events/events#cache-query-events > >>> > >>> - > >>> Denis > >>> > >>> > >>> On Fri, Jul 10, 2020 at 3:54 AM Max Timonin > >>> wrote: > >>> > Hi Igniters! > > We're going to protocol all input SQL queries from our users. > Currently > there is no such mechanism in Ignite to use for it. So we're proposing > >> to > add a new event: QUERY_EXECUITION_EVENT. > > Requirements for the event: > 1. If this event fires it means that a query is correct and will be > executed (and failed only in exceptional cases); > > 2. Event fires for all query types; > > 3. Required fields are: > - text of a query (with hidden arguments); > - arguments of query; > - query type; > - node id. > > Looks that this event should go along with `runningQryMgr::register` > in > class `IgniteH2Indexing` as this method invoked for all input queries > >>> too. > > What do you think? > > Regards, > Maksim > > >>> > >> > > -- Живи с улыбкой! :D
Re: Proposal of new event QUERY_EXECUTION_EVENT
Maksim, Can we change the EVT_CACHE_QUERY_EXECUTED to fire earlier? Or should there be an EVT_CACHE_QUERY_EXECUTION_STARTED for the query start, while the old event would continue to work for query finish? I think the new event needs to either reuse the old one, or be its mirror for the query start. It also means that we probably should resolve the issues you've listed. Would the event notification be synchronous? In which thread? Asynchronous is generally preferred - would it work? What happens in case the event listener fails? Thanks, Stan > On 16 Jul 2020, at 18:49, Denis Magda wrote: > > Taras, Yury, Ivan, > > Could you please join this thread and share your thoughts? Do we already > have any plans on tracking of the DDL and DML queries? > > - > Denis > > > On Wed, Jul 15, 2020 at 12:09 AM Max Timonin > wrote: > >> Hi Denis, thanks for the answer! >> >> We already checked EVT_CACHE_QUERY_EXECUTED and found that it works only in >> cases: >> 1. Scan queries and Select queries (common pattern is access to cache >> data); >> 2. This event triggers only if query execution succeeds, in case of failure >> while execution this event won't fire. >> >> Our additional requirements are to protocol queries: >> 1. that aren't cache related (for example, alter user); >> 2. that relate to multiple caches (while EVT_CACHE_QUERY_EXECUTED have >> field cacheName related to specific cache); >> 3. we need to protocol also DDL and DML queries. >> >> Regards, >> Maksim >> >> On Tue, Jul 14, 2020 at 10:20 PM Denis Magda wrote: >> >>> Hi Max, >>> >>> Could you check if the EVT_CACHE_QUERY_EXECUTED event is what you're >>> looking for? >>> >>> >> https://www.gridgain.com/docs/latest/developers-guide/events/events#cache-query-events >>> >>> - >>> Denis >>> >>> >>> On Fri, Jul 10, 2020 at 3:54 AM Max Timonin >>> wrote: >>> Hi Igniters! We're going to protocol all input SQL queries from our users. Currently there is no such mechanism in Ignite to use for it. So we're proposing >> to add a new event: QUERY_EXECUITION_EVENT. Requirements for the event: 1. If this event fires it means that a query is correct and will be executed (and failed only in exceptional cases); 2. Event fires for all query types; 3. Required fields are: - text of a query (with hidden arguments); - arguments of query; - query type; - node id. Looks that this event should go along with `runningQryMgr::register` in class `IgniteH2Indexing` as this method invoked for all input queries >>> too. What do you think? Regards, Maksim >>> >>
[jira] [Created] (IGNITE-13269) Waiting for completion of operations on indexes before cache stop
Kirill Tkalenko created IGNITE-13269:
Summary: Waiting for completion of operations on indexes before
cache stop
Key: IGNITE-13269
URL: https://issues.apache.org/jira/browse/IGNITE-13269
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Currently, there is no waiting for completion of operation on indexes when
cache is stopped. Because of this, there may be errors, for example, when
restarting the node:
{code:java}
Suppressed: java.lang.AssertionError: Release pinned page: FullPageId
[pageId=000206bfc352, effectivePageId=06bfc352, grpId=-782612924]
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.releaseFreePage(PageMemoryImpl.java:1902)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$PagePool.access$2100(PageMemoryImpl.java:1773)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$ClearSegmentRunnable.run(PageMemoryImpl.java:2878)
... 3 common frames omitted
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
