[MTCGA]: new failures in builds [5302554] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. *New Critical Failure in master MVCC Cache 7 https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_MvccCache7?branch=%3Cdefault%3E No changes in the build - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 06:01:36 14-05-2020
Re: Extensions for control.sh
Perfect idea to use this the tool for configuration and addition of extensions! - Denis On Wed, May 13, 2020 at 11:43 AM Denis Mekhanikov wrote: > Hi everyone! > > Control.sh is a command-line management tool that you can use to manage > your grid and check its vital parameters like topology version or > availability of baseline nodes. It has is good set of commands which are > suitable to work with vanilla Ignite. > > There is also a way to extend functionality of Ignite by implementing a > 3rd-party plugin or a module. Any plugin or external module should have > some kind of API to manage and monitor its activity. > If a command-line management command needs to be added, then the only > way to achieve that is to provide an additional script, separate from > control.sh. If you use multiple such plugins, then the set of required > tools may grow and lead to confusion, which script should be used to > configure which extension. Instead of doing that it would be convenient > for users to have ability to use the same script, but with an extended > set of options. It should make lifes of 3rd-party vendors easier. > > Currently many integrations and community-supported modules are being > moved outside of the core product: > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization > I think it makes sense to provide a possibility to configure extensions > using control.sh, since their number will grow over time, and some of > them will require some runtime configuration. > > What do you think? > > Denis > >
[DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).
Hello Igniters. Recently, master key rotation for Apache Ignite Transparent Data Encryption was implemented [1], but some security standards (PCI DSS at least) require rotation of all encryption keys [2]. Currently, encryption occurs when reading/writing pages to disk, cache encryption keys are stored in metastore. I'm going to contribute cache encryption key rotation and want to consult what is the best way to re-encrypting existing data, I see two different strategies. 1. In place re-encryption: Using the old key, sequentially read all the pages from the datastore, mark as dirty and log them into the WAL. After checkpoint pages will be stored to disk encrypted with the new key (as usual, along with updates). This strategy requires store the identifier (number) of the encryption key into the encrypted page. pros: - can work in the background with minimal performance impact (this impact can be managed). cons: - page duplication in the WAL may affect performance and historical rebalance. 2. Copy partition with re-encryption. This strategy is similar to partition snapshotting [3] - create partition copy encrypted with the new key and then replace the original partition file with the new one (see details [4]). pros: - should work faster than "in place" re-encryption. cons: - re-encryption in active cluster (and on unstable topology) can be difficult to implement. (See more detailed comparison [5]) Re-encryption of existing data is a long and rare procedure (It is recommended to change the key every 6 months, but at least once every 2 years). Thus, re-encryption can be implemented for maintenance mode (for example, on a stable topology in a read-only cluster) and in such case the approach with partition copying seems simpler and faster. So, what do you think - do we need "online" re-encryption and which of the proposed options is best suited for this? [1] https://issues.apache.org/jira/browse/IGNITE-12186 [2] https://www.pcisecuritystandards.org/documents/PCI_DSS_v3-2-1.pdf [3] https://cwiki.apache.org/confluence/display/IGNITE/IEP-43%3A+Cluster+snapshots#IEP-43:Clustersnapshots-Partitionscopystrategy [4] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Copywithre-encryptiondesign. [5] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Comparison
Re: Moving binary metadata to PDS storage folder
Maxim, This way we'll introduce a migration procedure between versions, which we currently don't have. Different Ignite versions are compatible by persistence storage. If we add a migration script, we need to decide, whether we need to run it every time when the version is upgraded, or only some specific versions are affected. I suggest having a procedure that will look for metadata in the work directory, and if it finds it there, then the node will use it. Otherwise the persistence directory is used. Denis On 13.05.2020 21:40, Maxim Muzafarov wrote: Folks, I think it's important to discuss the following question regarding this thread: Should we consider moving the migration procedure from the java production code to migration scripts? From my understanding, keeping all such things in java production source code has some disadvantages: 1. It executes only once at the migration stage. 2. It affects the complexity of the source code and code maintenance. 3. Node crash cases must be covered during the migration procedure. 4. It affects the production usage e.g. the process doesn't have the right access to the old directory (migration already completed) and will fail the node start. The right behavior from my point should be: 1. Change the default path of binary/marshaller directory to the new one. 2. Provide migration scripts for users. WDYT? On Wed, 13 May 2020 at 21:10, Denis Mekhanikov wrote: Sounds great! It happens pretty frequently that users migrate to a new version of Ignite and preserve persistence files only without caring too much about the work folder. But it turns out, that the work folder actually has some important stuff. This improvement should help with this issue. What's about in-memory mode? As far as I know, we write binary metadata to disk even when no persistence is configured. Do you plan to address it in any way? Denis On 12.05.2020 15:56, Sergey Antonov wrote: Hello Semyon, This is a good idea! вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин : Hello Semyon, This is a good and long-awaited improvement! Thank you for your efforts! Thanks, S. вт, 12 мая 2020 г. в 15:11, Данилов Семён : Hello! I would like to propose moving /binary_meta and /marshaller folders to the PDS folder. Motivation: data, directly related to the persistence, is stored outside the persistence dir, which can lead to various issues and also is not very convenient to use. In particular, with k8s, deployment disk that is attached to a container can not be accessed from other containers or outside of k8s. In case if support will need to drop persistence except data, there will be no way to recover due to binary metadata is required to process PDS files. I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) and a pull request(https://github.com/apache/ignite/pull/7792) that fixes the issue. In that PR I made the following: * store binary meta and marshaller data inside db/ folder * if binary meta of marshaller are found in "legacy" locations -- safely move them to new locations during the node startup Kind regards, Semyon Danilov.
Extensions for control.sh
Hi everyone! Control.sh is a command-line management tool that you can use to manage your grid and check its vital parameters like topology version or availability of baseline nodes. It has is good set of commands which are suitable to work with vanilla Ignite. There is also a way to extend functionality of Ignite by implementing a 3rd-party plugin or a module. Any plugin or external module should have some kind of API to manage and monitor its activity. If a command-line management command needs to be added, then the only way to achieve that is to provide an additional script, separate from control.sh. If you use multiple such plugins, then the set of required tools may grow and lead to confusion, which script should be used to configure which extension. Instead of doing that it would be convenient for users to have ability to use the same script, but with an extended set of options. It should make lifes of 3rd-party vendors easier. Currently many integrations and community-supported modules are being moved outside of the core product: https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization I think it makes sense to provide a possibility to configure extensions using control.sh, since their number will grow over time, and some of them will require some runtime configuration. What do you think? Denis
Re: Moving binary metadata to PDS storage folder
Folks, I think it's important to discuss the following question regarding this thread: Should we consider moving the migration procedure from the java production code to migration scripts? >From my understanding, keeping all such things in java production source code has some disadvantages: 1. It executes only once at the migration stage. 2. It affects the complexity of the source code and code maintenance. 3. Node crash cases must be covered during the migration procedure. 4. It affects the production usage e.g. the process doesn't have the right access to the old directory (migration already completed) and will fail the node start. The right behavior from my point should be: 1. Change the default path of binary/marshaller directory to the new one. 2. Provide migration scripts for users. WDYT? On Wed, 13 May 2020 at 21:10, Denis Mekhanikov wrote: > > Sounds great! > > It happens pretty frequently that users migrate to a new version of > Ignite and preserve persistence files only without caring too much about > the work folder. But it turns out, that the work folder actually has > some important stuff. > This improvement should help with this issue. > > What's about in-memory mode? As far as I know, we write binary metadata > to disk even when no persistence is configured. Do you plan to address > it in any way? > > Denis > > On 12.05.2020 15:56, Sergey Antonov wrote: > > Hello Semyon, > > > > This is a good idea! > > > > вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин : > > > >> Hello Semyon, > >> > >> This is a good and long-awaited improvement! Thank you for your efforts! > >> > >> Thanks, > >> S. > >> > >> вт, 12 мая 2020 г. в 15:11, Данилов Семён : > >> > >>> Hello! > >>> > >>> I would like to propose moving /binary_meta and /marshaller folders to > >> the > >>> PDS folder. > >>> > >>> Motivation: data, directly related to the persistence, is stored outside > >>> the persistence dir, which can lead to various issues and also is not > >> very > >>> convenient to use. In particular, with k8s, deployment disk that is > >>> attached to a container can not be accessed from other containers or > >>> outside of k8s. In case if support will need to drop persistence except > >>> data, there will be no way to recover due to binary metadata is required > >> to > >>> process PDS files. > >>> > >>> I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) > >> and a > >>> pull request(https://github.com/apache/ignite/pull/7792) that fixes the > >>> issue. > >>> > >>> In that PR I made the following: > >>> > >>> > >>> * store binary meta and marshaller data inside db/ folder > >>> * if binary meta of marshaller are found in "legacy" locations -- > >>> safely move them to new locations during the node startup > >>> > >>> > >>> Kind regards, > >>> > >>> Semyon Danilov. > >>> > >
Re: Moving binary metadata to PDS storage folder
Sounds great! It happens pretty frequently that users migrate to a new version of Ignite and preserve persistence files only without caring too much about the work folder. But it turns out, that the work folder actually has some important stuff. This improvement should help with this issue. What's about in-memory mode? As far as I know, we write binary metadata to disk even when no persistence is configured. Do you plan to address it in any way? Denis On 12.05.2020 15:56, Sergey Antonov wrote: Hello Semyon, This is a good idea! вт, 12 мая 2020 г. в 15:53, Вячеслав Коптилин : Hello Semyon, This is a good and long-awaited improvement! Thank you for your efforts! Thanks, S. вт, 12 мая 2020 г. в 15:11, Данилов Семён : Hello! I would like to propose moving /binary_meta and /marshaller folders to the PDS folder. Motivation: data, directly related to the persistence, is stored outside the persistence dir, which can lead to various issues and also is not very convenient to use. In particular, with k8s, deployment disk that is attached to a container can not be accessed from other containers or outside of k8s. In case if support will need to drop persistence except data, there will be no way to recover due to binary metadata is required to process PDS files. I created an issue (https://issues.apache.org/jira/browse/IGNITE-12994) and a pull request(https://github.com/apache/ignite/pull/7792) that fixes the issue. In that PR I made the following: * store binary meta and marshaller data inside db/ folder * if binary meta of marshaller are found in "legacy" locations -- safely move them to new locations during the node startup Kind regards, Semyon Danilov.
[jira] [Created] (IGNITE-13009) NoopTimeBag should be used if logging disabled
Anton Vinogradov created IGNITE-13009: - Summary: NoopTimeBag should be used if logging disabled Key: IGNITE-13009 URL: https://issues.apache.org/jira/browse/IGNITE-13009 Project: Ignite Issue Type: Task Reporter: Anton Vinogradov Assignee: Anton Vinogradov TimeBag uses performs useless operations on finishGlobalStage if logging disabled. Better case is to use no-op realization instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13008) Java thin client: Imlpement compatibility suite
Igor Sapego created IGNITE-13008: Summary: Java thin client: Imlpement compatibility suite Key: IGNITE-13008 URL: https://issues.apache.org/jira/browse/IGNITE-13008 Project: Ignite Issue Type: Improvement Components: thin client Reporter: Igor Sapego Fix For: 2.9 We need to implement a test suite which is going to cross-test new client with all previous versions of server, and new server with all released versions of client if possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Stop Maintenance of Ignite WebConsole
+1 Thanks, S. ср, 13 мая 2020 г. в 08:48, Nikolay Izhikov : > +1 > > ср, 13 мая 2020 г., 07:35 Alexey Zinoviev : > > > +1 > > > > ср, 13 мая 2020 г., 5:52 Saikat Maitra : > > > > > +1 > > > > > > Regards > > > Saikat > > > > > > On Tue, 12 May 2020 at 7:03 PM, Denis Magda wrote: > > > > > > > Igniters, > > > > > > > > As it was discussed earlier [1], it feels like we abandoned the > > > development > > > > and maintenance of Ignite WebConsole. The users keep creating tickets > > for > > > > improvements and report issues, while the backlog only keeps growing > > with > > > > no reaction on our end. > > > > > > > > With this vote, we hope to put WebConsole's development and > maintenance > > > on > > > > hold *officially* by moving it to a separate Ignite repository and > > > closing > > > > all WebConsole-related tickets with a proper maintenance > > discontinuation > > > > notice. That's a gentle phase-out. If anybody joins the community and > > > > decides to resurrect the project they will have a chance to do that. > > > > Otherwise, WebConsole ends up on a graveyard eventually. > > > > > > > > Please cast your vote: > > > > +1 - to discontinue Ignite WebConsole's development and maintenance > by > > > > moving to a separate Ignite repository. > > > > -1 - against this course of action. > > > > > > > > I'll let this vote last for 7 days and close it on May 18th. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Ignite-WebConsole-Deprecation-td47259.html > > > > > > > > - > > > > Denis > > > > > > > > > >
[jira] [Created] (IGNITE-13007) JDBC: Introduce feature flags for JDBC thin
Konstantin Orlov created IGNITE-13007: - Summary: JDBC: Introduce feature flags for JDBC thin Key: IGNITE-13007 URL: https://issues.apache.org/jira/browse/IGNITE-13007 Project: Ignite Issue Type: Improvement Components: sql Reporter: Konstantin Orlov Assignee: Konstantin Orlov Motivation the same as for https://issues.apache.org/jira/browse/IGNITE-12853 The thin client & JDBC, ODBC have different protocol specific and may require implement different features. Each client (thin cli, thin JDBC, ODBC) should have its own feature flags set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13006) Apache Ignite spring libs upgrade from version 4x to spring 5.2 version or later..
Devendra Jain created IGNITE-13006: -- Summary: Apache Ignite spring libs upgrade from version 4x to spring 5.2 version or later.. Key: IGNITE-13006 URL: https://issues.apache.org/jira/browse/IGNITE-13006 Project: Ignite Issue Type: Improvement Components: spring Affects Versions: 2.8 Environment: We are trying this on linux env. Reporter: Devendra Jain With Apache Ignite Release 2.8, Spring libs still using spring older version 4.3x. Do you have a plan to upgrade to Spring libs with 5.2x version or latter by end of this year. Since we are looking to use Apache ignite but due to some policy issue we can't be able to use Ignite 2.8 version which is still using older spring libarary version 4.3x. Please let us know as soon as possible. How you are planning to address this issue and what is the time new release including this fix. Thanks, Devendra Jain deven...@gmail.com -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-13005) Spring Data 2 - JPA Improvements and working with multiple Ignite instances on same JVM
Manuel Núñez created IGNITE-13005:
-
Summary: Spring Data 2 - JPA Improvements and working with
multiple Ignite instances on same JVM
Key: IGNITE-13005
URL: https://issues.apache.org/jira/browse/IGNITE-13005
Project: Ignite
Issue Type: Improvement
Components: spring
Affects Versions: 2.7.6
Reporter: Manuel Núñez
Assignee: Ilya Kasnacheev
I have it working for Spring Data 2 (2.7.6) module with some interesting
improvements, but by now I don't have enough time to give it the attention it
requires, full unit/integration tests..., sorry a lot. ¿maybe any of you have
the time?. Thanks community!!
Code is 100% compatible with previous versions.
[https://github.com/hawkore/ignite-hk/tree/master/modules/spring-data-2.0]
* Supports multiple ignite instances on same JVM (@RepositoryConfig).
* Supports query tuning parameters in {{@Query}} annotation
* Supports projections
* Supports {{Page}} and {{Stream}} responses
* Supports Sql Fields Query resultset transformation into the domain entity
* Supports named parameters ({{:myParam}}) into SQL queries, declared using
{{@Param("myParam")}}
* Supports advanced parameter binding and SpEL expressions into SQL queries:
** *Template variables*:
*** {{#entityName}} - the simple class name of the domain entity
** *Method parameter expressions*: Parameters are exposed for indexed access
({{[0]}} is the first query method's param) or via the name declared using
{{@Param}}. The actual SpEL expression binding is triggered by {{?#}}. Example:
{{?#{[0]}}} or {{?#{#myParamName}}}
** *Advanced SpEL expressions*: While advanced parameter binding is a very
useful feature, the real power of SpEL stems from the fact, that the
expressions can refer to framework abstractions or other application components
through SpEL EvaluationContext extension model.
* Supports SpEL expressions into Text queries ({{TextQuery}}). Examples:
{code:java}
@Query(value = "SELECT * from #{#entityName} where email = :email")
User searchUserByEmail(@Param("email") String email);
@Query(value = "SELECT * from #{#entityName} where country = ?#{[0] and city =
?#{[1]}")
List searchUsersByCity(@Param("country") String country, @Param("city")
String city, Pageable pageable);
@Query(value = "SELECT * from #{#entityName} where email = ?")
User searchUserByEmail(String email);
@Query(value = "SELECT * from #{#entityName} where uuidCity = ?#{
mySpELFunctionsBean.cityNameToUUID(#city)}")
List searchUsersByCity(@Param("city") String city, Pageable pageable);
@Query(textQuery = true, value = "email: #{#email}")
User searchUserByEmail(@Param("email") String email);
@Query(textQuery = true, value = "#{#textToSearch}")
List searchUsersByText(@Param("textToSearch") String text, Pageable
pageable);
@Query(textQuery = true, value = "#{[0]}")
List searchUsersByText(String textToSearch, Pageable pageable);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
