[GitHub] ignite pull request #5468: IGNITE-10374 if rebalance isn't restarted no need...
GitHub user sergey-chugunov-1985 opened a pull request: https://github.com/apache/ignite/pull/5468 IGNITE-10374 if rebalance isn't restarted no need to disable already ⦠â¦disabled WAL You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10374 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5468.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5468 commit 22d0ac9f0c9ae16bb9c1ce24c8c6f05b586e1073 Author: Sergey Chugunov Date: 2018-11-22T07:56:44Z IGNITE-10374 if rebalance isn't restarted no need to disable already disabled WAL ---
Re: New API for changing configuration of persistent caches
My variant of API avoids cache configuration. One more thing to note - as we found out control.sh cannot dump XML configuration. Currently it returns only subset of properties. And in general case it is impossible to convert CacheConfiguration to Spring XML, because Spring XMLis not serialization protocol. So API with CacheConfiguration doesn’t seem to work for control.sh as well. чт, 22 нояб. 2018 г. в 10:05, Eduard Shangareev : > Vovan, > > We couldn't avoid API with cache configuration. > Almost all of ~70 properties could be changed, some of them are instances > of objects or could be user-defined class. > Could you come up with alternatives for user-defined affinity function? > > Also, the race would have a place in other scenarios. > > > > On Thu, Nov 22, 2018 at 8:50 AM Vladimir Ozerov > wrote: > > > Ed, > > > > We may have API similar to “cache” and “getOrCreateCache”, or may not. It > > is up to us to decide. Similarity on it’s own is weak argument. > > Functionality and simplicity - this is what matters. > > > > Approach with cache configuration has three major issues > > 1) It exposes properties which user will not be able to change, so > typical > > user actions would be: try to change property, fail as it is unsupported, > > go reading documentation. Approach with separate POJO is intuitive and > > self-documenting. > > 2) It has race condition between config read and config apply, so user do > > not know what exactly he changes, unless you change API to something like > > “restartCaches(Tuple...)”, which > > user will need to call in a loop. > > 3) And it is not suitable for non-Java platform, which is a showstopper - > > all API should be available from all platforms unless it is proven to be > > impossible to implement. > > > > Vladimir. > > > > чт, 22 нояб. 2018 г. в 1:06, Eduard Shangareev < > > eduard.shangar...@gmail.com > > >: > > > > > Vovan, > > > > > > Would you argue that we should have the similar API in Java as > > > Ignite.cache(CacheConfiguration) or > > > Ignite.getOrCreateCache(CacheConfiguration)? > > > > > > With a proposed solution, every other API call would rely on it > finally. > > > > > > I am interested in having such feature not arguing about API > > alternatives. > > > > > > We definitely should have the ability to change it via control.sh and > > Java > > > API. Everything else is optional from my point of view (at least on the > > > current stage). > > > > > > Moreover, your arguments are more about our format of > CacheConfiguration > > > which couldn't be defined in other languages and clients. So, maybe we > > > should start a discussion about how we should change it in 3.0? > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 7:45 PM Vladimir Ozerov > > > wrote: > > > > > > > Ed, > > > > > > > > Why do we want to operate on CacheConfiguration so desperately? Your > > > > example raises even more questions: > > > > 1) What to do with thin clients? > > > > 2) What to do with aforementioned race conditions, when cache could > be > > > > changed concurrently? > > > > 3) Why such trivial operation from user perspective is only supported > > > from > > > > control.sh and not from the rest API (even Java client nodes will be > > > > affected - remember our plans to remove requirement to have cache > > classes > > > > on client nodes, which is yet to be implemented). > > > > > > > > Compare it to alternative API: > > > > > > > > 1) Native call from any language without limitations: > > > > > > > > > > > > > > Ignite.changeCache(CacheConfigurationChange.create().setCacheMode(REPLICATED).setBackups(2)); > > > > > > > > 2) Call from control.sh in one line without race conditions with > > > concurrent > > > > cache changes: > > > > control.sh --cache --change cacheMode=REPLICATED backups=2 > > > > > > > > On Wed, Nov 21, 2018 at 7:32 PM Eduard Shangareev < > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > Vovan, > > > > > > > > > > user already is able to get cache configuration as xml. > > > > > control.sh --cache list '.' --config > > > > > > > > > > So, user could update it and run: > > > > > control.sh --cache --restart -cfg=xml.path > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 7:06 PM Vladimir Ozerov < > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Ed, > > > > > > > > > > > > He can do that programmatically. But I meant another case - Java > > node > > > > > > creates a cache. Then .NET node wants to change it. Proposed API > > > cannot > > > > > > handle it. > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 19:03, Eduard Shangareev < > > > > > eshangar...@gridgain.com > > > > > > >: > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > I didn't get how does .Net user start caches right now? XML and > > > > remote > > > > > > > node? Right? > > > > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:55 PM Vladimir Ozerov < > > > > voze...@gridgain.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Ed, >
[jira] [Created] (IGNITE-10374) Node doesn't own rebalanced partitions on rebalancing finished
Sergey Chugunov created IGNITE-10374: Summary: Node doesn't own rebalanced partitions on rebalancing finished Key: IGNITE-10374 URL: https://issues.apache.org/jira/browse/IGNITE-10374 Project: Ignite Issue Type: Bug Reporter: Sergey Chugunov Assignee: Sergey Chugunov Fix For: 2.8 Prerequisite: flag *IGNITE_DISABLE_WAL_DURING_REBALANCING* is set to true (default value is false). Scenario: * Node joins the grid and starts rebalancing all cache groups from scratch (e.g. all db files of the node were cleaned up during its downtime); * One or more client nodes join topology when rebalancing is in progress. Expected outcome: Rebalance finishes, node owns all received partitions, new affinity is applied. Actual outcome: Rebalance finishes, but node doesn't own any of received partitions, no affinity changes take place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: New API for changing configuration of persistent caches
Vovan, We couldn't avoid API with cache configuration. Almost all of ~70 properties could be changed, some of them are instances of objects or could be user-defined class. Could you come up with alternatives for user-defined affinity function? Also, the race would have a place in other scenarios. On Thu, Nov 22, 2018 at 8:50 AM Vladimir Ozerov wrote: > Ed, > > We may have API similar to “cache” and “getOrCreateCache”, or may not. It > is up to us to decide. Similarity on it’s own is weak argument. > Functionality and simplicity - this is what matters. > > Approach with cache configuration has three major issues > 1) It exposes properties which user will not be able to change, so typical > user actions would be: try to change property, fail as it is unsupported, > go reading documentation. Approach with separate POJO is intuitive and > self-documenting. > 2) It has race condition between config read and config apply, so user do > not know what exactly he changes, unless you change API to something like > “restartCaches(Tuple...)”, which > user will need to call in a loop. > 3) And it is not suitable for non-Java platform, which is a showstopper - > all API should be available from all platforms unless it is proven to be > impossible to implement. > > Vladimir. > > чт, 22 нояб. 2018 г. в 1:06, Eduard Shangareev < > eduard.shangar...@gmail.com > >: > > > Vovan, > > > > Would you argue that we should have the similar API in Java as > > Ignite.cache(CacheConfiguration) or > > Ignite.getOrCreateCache(CacheConfiguration)? > > > > With a proposed solution, every other API call would rely on it finally. > > > > I am interested in having such feature not arguing about API > alternatives. > > > > We definitely should have the ability to change it via control.sh and > Java > > API. Everything else is optional from my point of view (at least on the > > current stage). > > > > Moreover, your arguments are more about our format of CacheConfiguration > > which couldn't be defined in other languages and clients. So, maybe we > > should start a discussion about how we should change it in 3.0? > > > > > > > > > > On Wed, Nov 21, 2018 at 7:45 PM Vladimir Ozerov > > wrote: > > > > > Ed, > > > > > > Why do we want to operate on CacheConfiguration so desperately? Your > > > example raises even more questions: > > > 1) What to do with thin clients? > > > 2) What to do with aforementioned race conditions, when cache could be > > > changed concurrently? > > > 3) Why such trivial operation from user perspective is only supported > > from > > > control.sh and not from the rest API (even Java client nodes will be > > > affected - remember our plans to remove requirement to have cache > classes > > > on client nodes, which is yet to be implemented). > > > > > > Compare it to alternative API: > > > > > > 1) Native call from any language without limitations: > > > > > > > > > Ignite.changeCache(CacheConfigurationChange.create().setCacheMode(REPLICATED).setBackups(2)); > > > > > > 2) Call from control.sh in one line without race conditions with > > concurrent > > > cache changes: > > > control.sh --cache --change cacheMode=REPLICATED backups=2 > > > > > > On Wed, Nov 21, 2018 at 7:32 PM Eduard Shangareev < > > > eduard.shangar...@gmail.com> wrote: > > > > > > > Vovan, > > > > > > > > user already is able to get cache configuration as xml. > > > > control.sh --cache list '.' --config > > > > > > > > So, user could update it and run: > > > > control.sh --cache --restart -cfg=xml.path > > > > > > > > > > > > On Wed, Nov 21, 2018 at 7:06 PM Vladimir Ozerov < > voze...@gridgain.com> > > > > wrote: > > > > > > > > > Ed, > > > > > > > > > > He can do that programmatically. But I meant another case - Java > node > > > > > creates a cache. Then .NET node wants to change it. Proposed API > > cannot > > > > > handle it. > > > > > > > > > > ср, 21 нояб. 2018 г. в 19:03, Eduard Shangareev < > > > > eshangar...@gridgain.com > > > > > >: > > > > > > > > > > > Vladimir, > > > > > > > > > > > > I didn't get how does .Net user start caches right now? XML and > > > remote > > > > > > node? Right? > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:55 PM Vladimir Ozerov < > > > voze...@gridgain.com> > > > > > > wrote: > > > > > > > > > > > > > Ed, > > > > > > > > > > > > > > We are not Java product. We support 6 platforms at the moment. > > Why > > > do > > > > > we > > > > > > > implement a feature which can only be used in Java, when it is > > very > > > > > easy > > > > > > to > > > > > > > make it available from everywhere? > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:50, Eduard Shangareev < > > > > > > eshangar...@gridgain.com > > > > > > > >: > > > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > > > It would be Java API specific. > > > > > > > > For a user, we would add a new command for console.sh which > > would > > > > > take > > > > > > an > > > > > > > > XML-file path as a parameter. > > > > > > > > > > > >
Re: [IMPORTANT] Future of Binary Objects
It may be OK to extend compatible field types (like from Int to Long).
In Protobuf for example this is allowed just because there is no difference
between Int and Long in binary format: they all are equally varlen encoded
and Longs just will occupy up to 9 bytes, while Ints up to 5.
But for every other case, where binary representation is type dependent, I
would be against. This will either require to migrate the whole dataset to
a new model (which is always risky, since you may need to rollback to
previous version of your code) or it will require type checks/conversions
for each field access, which is a hard to reason complication and possible
performance penalty.
Sergi
чт, 22 нояб. 2018 г. в 09:23, Vladimir Ozerov :
> Denis,
>
> Several examples:
> 1) DEFAULT values - in SQL you may avoid storing default value in the table
> and store it in metadata instead. Not applicable for BinaryObject because
> the same binary object may be saved to two SQL tables with different
> defaults
> 2) DATE and other temporal types - in SQL you want to store it in special
> format to be able to extract date parts quickly (typically - 11 bytes). But
> in Java and some other languages the best format is plain long. this is why
> we use it BinaryObject
> 3) String charset - in SQL you may choose different charsets for different
> tables. E.g. UTF-8 for one, ASCII for another. In BinaryObject we store
> everything in UTF-8, and this is fine for most cases, well ... except of
> SQL :-)
>
> The key thing here is that you cannot define a format which will be good
> for both SQL, and native API. They are very different. This is why I
> propose to define additional interface on cache level defining how to store
> values, which will be very different from binary objects.
>
> Vladimir.
>
> On Thu, Nov 22, 2018 at 3:32 AM Denis Magda wrote:
>
> > Vladimir,
> >
> > Could you educate me a little bit, why the current format is bad for SQL
> > and why another one is more suitable?
> >
> > Also, if we introduce the new format then why would we keep the binary
> one?
> > Is the new format just a next version of the binary one.
> >
> > 2.3) Remove restrictions on changing field type
> > > I do not know why we did that in the first place. This restriction
> > prevents
> > > type evolution and confuses users.
> >
> >
> > That is a hot requirement shared by those who use Ignite SQL in
> production.
> > +1.
> >
> > --
> > Denis
> >
> > On Mon, Nov 19, 2018 at 11:05 PM Vladimir Ozerov
> > wrote:
> >
> > > Igniters,
> > >
> > > It is very likely that Apache Ignite 3.0 will be released next year. So
> > we
> > > need to start thinking about major product improvements. I'd like to
> > start
> > > with binary objects.
> > >
> > > Currently they are one of the main limiting factors for the product.
> They
> > > are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> > > comparing to other vendors. They are slow - not suitable for SQL at
> all.
> > >
> > > I would like to ask all of you who worked with binary objects to share
> > your
> > > feedback and ideas, so that we understand how they should look like in
> AI
> > > 3.0. This is a brain storm - let's accumulate ideas first and minimize
> > > critics. Then we will work on ideas in separate topics.
> > >
> > > 1) Historical background
> > >
> > > BO were implemented around 2014 (Apache Ignite 1.5) when we started
> > working
> > > on .NET and CPP clients. During design we had several ideas in mind:
> > > - ability to read object fields in O(1) without deserialization
> > > - interoperabillty between Java, .NET and CPP.
> > >
> > > Since then a number of other concepts were mixed to the cocktail:
> > > - Affinity key fields
> > > - Strict typing for existing fields (aka metadata)
> > > - Binary Object as storage format
> > >
> > > 2) My proposals
> > >
> > > 2.1) Introduce "Data Row Format" interface
> > > Binary Objects are terrible candidates for storage. Too fat, too slow.
> > > Efficient storage typically has <10 bytes overhead per row (no
> metadata,
> > no
> > > length, no hash code, etc), allow supper-fast field access, support
> > > different string formats (ASCII, UTF-8, etc), support different
> temporal
> > > types (date, time, timestamp, timestamp with timezone, etc), and store
> > > these types as efficiently as possible.
> > >
> > > What we need is to introduce an interface which will convert a pair of
> > > key-value objects into a row. This row will be used to store data and
> to
> > > get fields from it. Care about memory consumption, need SQL and strict
> > > schema - use one format. Need flexibility and prefer key-value access -
> > use
> > > another format which will store binary objects unchanged (current
> > > behavior).
> > >
> > > interface DataRowFormat {
> > > DataRow create(Object key, Object value); // primitives or binary
> > > objects
> > > DataRowMetadata metadata();
> > > }
> > >
> > > 2.2) Remove affinity field from metadata
> > > Affinity
Re: [IMPORTANT] Future of Binary Objects
Val,
If we treat binary object as a plain container of fields with certain names
and types, we do not care how to convert Int to String. This is up to user
to decide how to migrate.
Ignite could help users in some cases. E.g. for SQL caches we may provide
ALTER TABLE command, which will do necessary conversions on storage layer.
On Thu, Nov 22, 2018 at 4:27 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:
> We should definitely allow to change type of field/column to another
> compatible type. The fact that we do not allow to change Int to Long is
> pretty insane. However, there are cases when it's much more complicated.
> How are we going to replace Int with a String, for example? I believe this
> should require certain migration procedure anyway. How do other databases
> handle that?
>
> -Val
>
> On Wed, Nov 21, 2018 at 4:32 PM Denis Magda wrote:
>
> > Vladimir,
> >
> > Could you educate me a little bit, why the current format is bad for SQL
> > and why another one is more suitable?
> >
> > Also, if we introduce the new format then why would we keep the binary
> one?
> > Is the new format just a next version of the binary one.
> >
> > 2.3) Remove restrictions on changing field type
> > > I do not know why we did that in the first place. This restriction
> > prevents
> > > type evolution and confuses users.
> >
> >
> > That is a hot requirement shared by those who use Ignite SQL in
> production.
> > +1.
> >
> > --
> > Denis
> >
> > On Mon, Nov 19, 2018 at 11:05 PM Vladimir Ozerov
> > wrote:
> >
> > > Igniters,
> > >
> > > It is very likely that Apache Ignite 3.0 will be released next year. So
> > we
> > > need to start thinking about major product improvements. I'd like to
> > start
> > > with binary objects.
> > >
> > > Currently they are one of the main limiting factors for the product.
> They
> > > are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> > > comparing to other vendors. They are slow - not suitable for SQL at
> all.
> > >
> > > I would like to ask all of you who worked with binary objects to share
> > your
> > > feedback and ideas, so that we understand how they should look like in
> AI
> > > 3.0. This is a brain storm - let's accumulate ideas first and minimize
> > > critics. Then we will work on ideas in separate topics.
> > >
> > > 1) Historical background
> > >
> > > BO were implemented around 2014 (Apache Ignite 1.5) when we started
> > working
> > > on .NET and CPP clients. During design we had several ideas in mind:
> > > - ability to read object fields in O(1) without deserialization
> > > - interoperabillty between Java, .NET and CPP.
> > >
> > > Since then a number of other concepts were mixed to the cocktail:
> > > - Affinity key fields
> > > - Strict typing for existing fields (aka metadata)
> > > - Binary Object as storage format
> > >
> > > 2) My proposals
> > >
> > > 2.1) Introduce "Data Row Format" interface
> > > Binary Objects are terrible candidates for storage. Too fat, too slow.
> > > Efficient storage typically has <10 bytes overhead per row (no
> metadata,
> > no
> > > length, no hash code, etc), allow supper-fast field access, support
> > > different string formats (ASCII, UTF-8, etc), support different
> temporal
> > > types (date, time, timestamp, timestamp with timezone, etc), and store
> > > these types as efficiently as possible.
> > >
> > > What we need is to introduce an interface which will convert a pair of
> > > key-value objects into a row. This row will be used to store data and
> to
> > > get fields from it. Care about memory consumption, need SQL and strict
> > > schema - use one format. Need flexibility and prefer key-value access -
> > use
> > > another format which will store binary objects unchanged (current
> > > behavior).
> > >
> > > interface DataRowFormat {
> > > DataRow create(Object key, Object value); // primitives or binary
> > > objects
> > > DataRowMetadata metadata();
> > > }
> > >
> > > 2.2) Remove affinity field from metadata
> > > Affinity rules are governed by cache, not type. We should remove
> > > "affintiyFieldName" from metadata.
> > >
> > > 2.3) Remove restrictions on changing field type
> > > I do not know why we did that in the first place. This restriction
> > prevents
> > > type evolution and confuses users.
> > >
> > > 2.4) Use bitmaps for "null" and default values and for fixed-length
> > fields,
> > > put fixed-length fields before variable-length.
> > > Motivation: to save space.
> > >
> > > What else? Please share your ideas.
> > >
> > > Vladimir.
> > >
> >
>
[jira] [Created] (IGNITE-10373) Web console: we need to fix docker-compose.yml
Pavel Konstantinov created IGNITE-10373:
---
Summary: Web console: we need to fix docker-compose.yml
Key: IGNITE-10373
URL: https://issues.apache.org/jira/browse/IGNITE-10373
Project: Ignite
Issue Type: Bug
Components: wizards
Reporter: Pavel Konstantinov
It located here 'modules/web-console/docker/compose' and it doesn't work
{code}
Version in "./docker-compose.yml" is invalid - it should be a string.
{code}
after I've fixed that I've got the next one
{code}
Version in "./docker-compose.yml" is invalid. You might be seeing this error
because you're using the wrong Compose file version. Either specify a supported
version (e.g "2.2" or "3.3") and place your service definitions under the
`services` key, or omit the `version` key and place your service definitions at
the root of the file to use version 1.
{code}
after I've fixed that I've got the next one
{code}
ERROR: Service 'frontend' depends on service 'testenv' which is undefined.
{code}
So, please make it workable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [IMPORTANT] Future of Binary Objects
Denis,
Several examples:
1) DEFAULT values - in SQL you may avoid storing default value in the table
and store it in metadata instead. Not applicable for BinaryObject because
the same binary object may be saved to two SQL tables with different
defaults
2) DATE and other temporal types - in SQL you want to store it in special
format to be able to extract date parts quickly (typically - 11 bytes). But
in Java and some other languages the best format is plain long. this is why
we use it BinaryObject
3) String charset - in SQL you may choose different charsets for different
tables. E.g. UTF-8 for one, ASCII for another. In BinaryObject we store
everything in UTF-8, and this is fine for most cases, well ... except of
SQL :-)
The key thing here is that you cannot define a format which will be good
for both SQL, and native API. They are very different. This is why I
propose to define additional interface on cache level defining how to store
values, which will be very different from binary objects.
Vladimir.
On Thu, Nov 22, 2018 at 3:32 AM Denis Magda wrote:
> Vladimir,
>
> Could you educate me a little bit, why the current format is bad for SQL
> and why another one is more suitable?
>
> Also, if we introduce the new format then why would we keep the binary one?
> Is the new format just a next version of the binary one.
>
> 2.3) Remove restrictions on changing field type
> > I do not know why we did that in the first place. This restriction
> prevents
> > type evolution and confuses users.
>
>
> That is a hot requirement shared by those who use Ignite SQL in production.
> +1.
>
> --
> Denis
>
> On Mon, Nov 19, 2018 at 11:05 PM Vladimir Ozerov
> wrote:
>
> > Igniters,
> >
> > It is very likely that Apache Ignite 3.0 will be released next year. So
> we
> > need to start thinking about major product improvements. I'd like to
> start
> > with binary objects.
> >
> > Currently they are one of the main limiting factors for the product. They
> > are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> > comparing to other vendors. They are slow - not suitable for SQL at all.
> >
> > I would like to ask all of you who worked with binary objects to share
> your
> > feedback and ideas, so that we understand how they should look like in AI
> > 3.0. This is a brain storm - let's accumulate ideas first and minimize
> > critics. Then we will work on ideas in separate topics.
> >
> > 1) Historical background
> >
> > BO were implemented around 2014 (Apache Ignite 1.5) when we started
> working
> > on .NET and CPP clients. During design we had several ideas in mind:
> > - ability to read object fields in O(1) without deserialization
> > - interoperabillty between Java, .NET and CPP.
> >
> > Since then a number of other concepts were mixed to the cocktail:
> > - Affinity key fields
> > - Strict typing for existing fields (aka metadata)
> > - Binary Object as storage format
> >
> > 2) My proposals
> >
> > 2.1) Introduce "Data Row Format" interface
> > Binary Objects are terrible candidates for storage. Too fat, too slow.
> > Efficient storage typically has <10 bytes overhead per row (no metadata,
> no
> > length, no hash code, etc), allow supper-fast field access, support
> > different string formats (ASCII, UTF-8, etc), support different temporal
> > types (date, time, timestamp, timestamp with timezone, etc), and store
> > these types as efficiently as possible.
> >
> > What we need is to introduce an interface which will convert a pair of
> > key-value objects into a row. This row will be used to store data and to
> > get fields from it. Care about memory consumption, need SQL and strict
> > schema - use one format. Need flexibility and prefer key-value access -
> use
> > another format which will store binary objects unchanged (current
> > behavior).
> >
> > interface DataRowFormat {
> > DataRow create(Object key, Object value); // primitives or binary
> > objects
> > DataRowMetadata metadata();
> > }
> >
> > 2.2) Remove affinity field from metadata
> > Affinity rules are governed by cache, not type. We should remove
> > "affintiyFieldName" from metadata.
> >
> > 2.3) Remove restrictions on changing field type
> > I do not know why we did that in the first place. This restriction
> prevents
> > type evolution and confuses users.
> >
> > 2.4) Use bitmaps for "null" and default values and for fixed-length
> fields,
> > put fixed-length fields before variable-length.
> > Motivation: to save space.
> >
> > What else? Please share your ideas.
> >
> > Vladimir.
> >
>
Re: [IMPORTANT] Future of Binary Objects
Ilya,
Currently binary objects already works almost as you proposed. We have 4
bytes types (type name hash) and we have 4 bytes schema ID (hash of all
field names). We do not write field IDs in the object itself. What we do
not have is separation of fixed and varlen fields. Agree, that we should
implement it and remove offsets of fixed fields from the binary object.
On Wed, Nov 21, 2018 at 7:18 PM Ilya Kasnacheev
wrote:
> Hello!
>
> I would like to propose the following changes:
>
> - Let's allow multiple BinaryType's per Class. Make typeId = cksum(list of
> class types + fields) as opposed of cksum(class name) as we have it
> currently. Note that we only have to compute that once per class loaded in
> JVM.
> - BinaryType has a list of fixed length fields (numbers, datetimes, flags)
> and list of variable length fields. We can put all fixed length fields at
> start of BinaryObject so that we can access them by offset as per typeId.
> - Likewise we don't need to encode field id in BinaryObject anymore, save 4
> bytes per field. We already know their order from BinaryType.
> - This means when you ALTER TABLE we add a BinaryType to existing Class (or
> pseudo-Class type name) and we can use it for new data, and eventually
> update existing data to have this field.
> - On top of BinaryType's we can have checks that run them against SQL table
> columns list to see if there are any mismatches.
>
> To Illustrate, previously we had it like:
> [ Type id | String field id | String field value | Long field id | Long
> field value | Datetime field id | Datetime field value ]
> But now it will be
> [ Type id | Long field value | Datetime field value | String field value ]
> ^--^ can be accessed by offset
>
> Regards,
> Ilya.
>
> --
> Ilya Kasnacheev
>
>
> вт, 20 нояб. 2018 г. в 10:05, Vladimir Ozerov :
>
> > Igniters,
> >
> > It is very likely that Apache Ignite 3.0 will be released next year. So
> we
> > need to start thinking about major product improvements. I'd like to
> start
> > with binary objects.
> >
> > Currently they are one of the main limiting factors for the product. They
> > are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> > comparing to other vendors. They are slow - not suitable for SQL at all.
> >
> > I would like to ask all of you who worked with binary objects to share
> your
> > feedback and ideas, so that we understand how they should look like in AI
> > 3.0. This is a brain storm - let's accumulate ideas first and minimize
> > critics. Then we will work on ideas in separate topics.
> >
> > 1) Historical background
> >
> > BO were implemented around 2014 (Apache Ignite 1.5) when we started
> working
> > on .NET and CPP clients. During design we had several ideas in mind:
> > - ability to read object fields in O(1) without deserialization
> > - interoperabillty between Java, .NET and CPP.
> >
> > Since then a number of other concepts were mixed to the cocktail:
> > - Affinity key fields
> > - Strict typing for existing fields (aka metadata)
> > - Binary Object as storage format
> >
> > 2) My proposals
> >
> > 2.1) Introduce "Data Row Format" interface
> > Binary Objects are terrible candidates for storage. Too fat, too slow.
> > Efficient storage typically has <10 bytes overhead per row (no metadata,
> no
> > length, no hash code, etc), allow supper-fast field access, support
> > different string formats (ASCII, UTF-8, etc), support different temporal
> > types (date, time, timestamp, timestamp with timezone, etc), and store
> > these types as efficiently as possible.
> >
> > What we need is to introduce an interface which will convert a pair of
> > key-value objects into a row. This row will be used to store data and to
> > get fields from it. Care about memory consumption, need SQL and strict
> > schema - use one format. Need flexibility and prefer key-value access -
> use
> > another format which will store binary objects unchanged (current
> > behavior).
> >
> > interface DataRowFormat {
> > DataRow create(Object key, Object value); // primitives or binary
> > objects
> > DataRowMetadata metadata();
> > }
> >
> > 2.2) Remove affinity field from metadata
> > Affinity rules are governed by cache, not type. We should remove
> > "affintiyFieldName" from metadata.
> >
> > 2.3) Remove restrictions on changing field type
> > I do not know why we did that in the first place. This restriction
> prevents
> > type evolution and confuses users.
> >
> > 2.4) Use bitmaps for "null" and default values and for fixed-length
> fields,
> > put fixed-length fields before variable-length.
> > Motivation: to save space.
> >
> > What else? Please share your ideas.
> >
> > Vladimir.
> >
>
Re: proposed realization KILL QUERY command
Denis, Problems with separate parameters are explained above. чт, 22 нояб. 2018 г. в 3:23, Denis Magda : > Vladimir, > > All of the alternatives are reminiscent of mathematical operations. Don't > look like a SQL command. What if we use a SQL approach introducing named > parameters: > > KILL QUERY query_id=10 [AND node_id=5] > > -- > Denis > > On Wed, Nov 21, 2018 at 4:11 AM Vladimir Ozerov > wrote: > > > Denis, > > > > Space is bad candidate because it is a whitespace. Without whitespaces we > > can have syntax without quotes at all. Any non-whitespace delimiter will > > work, though: > > > > KILL QUERY 45.1 > > KILL QUERY 45-1 > > KILL QUERY 45:1 > > > > On Wed, Nov 21, 2018 at 3:06 PM Юрий > wrote: > > > > > Denis, > > > > > > Let's consider parameter of KILL QUERY just a string with some query > id, > > > without any meaning for user. User just need to get the id and pass as > > > parameter to KILL QUERY command. > > > > > > Even if query is distributed it have single query id from user > > perspective > > > and will killed on all nodes. User just need to known one global query > > id. > > > > > > How it can works. > > > 1)SELECT * from running_queries > > > result is > > > query_id | node_id > > > | sql | schema_name | connection_id | duration > > > 123.33 | e0a69cb8-a1a8-45f6-b84d-ead367a0 | SELECT ... | ... > > > | 22 | 23456 > > > 333.31 | aaa6acb8-a4a5-42f6-f842-ead111b00020 | UPDATE... | > ... > > > | 321| 346 > > > 2) KILL QUERY '123.33' > > > > > > So, user need select query_id from running_queries view and use it for > > KILL > > > QUERY command. > > > > > > I hope it became clearer. > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 02:11, Denis Magda : > > > > > > > Folks, > > > > > > > > The decimal syntax is really odd - KILL QUERY > > > > '[node_order].[query_counter]' > > > > > > > > Confusing, let's use a space to separate parameters. > > > > > > > > Also, what if I want to halt a specific query with certain ID? Don't > > know > > > > the node number, just know that the query is distributed and runs > > across > > > > several machines. Sounds like the syntax still should consider > > > > [node_order/id] as an optional parameter. > > > > > > > > Probably, if you explain to me how an end user will use this command > > from > > > > the very beginning (how do I look for a query id and node id, etc) > then > > > the > > > > things get clearer. > > > > > > > > -- > > > > Denis > > > > > > > > On Tue, Nov 20, 2018 at 1:03 AM Юрий > > > wrote: > > > > > > > > > Hi Vladimir, > > > > > > > > > > Thanks for your suggestion to use MANAGEMENT_POOL for processing > > > > > cancellation requests. > > > > > > > > > > About your questions. > > > > > 1) I'm going to implements SQL view to provide list of running > > queries. > > > > The > > > > > SQL VIEW has been a little bit discussed earlier. Proposed name is > > > > > *running_queries* with following columns: query_id, node_id, sql, > > > > > schema_name, connection_id, duration. Currently most of the > > information > > > > can > > > > > be retrieved through cache API, however it doesn't matter, any > case > > we > > > > > need to expose SQL VIEW. Seem's you are right - the part should be > > > > > implemented firstly. > > > > > 2) Fully agree that we need to support all kind of SQL queries > > > > > (SLECT/DML/DDL, transactional, non transnational, local, > > distributed). > > > I > > > > > definitely sure that it will possible for all of above, however I'm > > not > > > > > sure about DDL - need to investigate it deeper. Also need to > > understand > > > > > that canceled DML operation can lead to partially updated data for > > non > > > > > transational caches. > > > > > > > > > > > > > > > > > > > > пн, 19 нояб. 2018 г. в 19:17, Vladimir Ozerov < > voze...@gridgain.com > > >: > > > > > > > > > > > Hi Yuriy, > > > > > > > > > > > > I think we can use MANAGEMENT_POOL for this. It is already used > for > > > > some > > > > > > internal Ignite tasks, and it appears to be a good candidate to > > > process > > > > > > cancel requests. > > > > > > > > > > > > But there are several things which are not clear enough for me at > > the > > > > > > moment: > > > > > > 1) How user is going to get the list of running queries in the > > first > > > > > place? > > > > > > Do we already have any SQL commands/views to get this > information? > > > > > > 2) We need to ensure that KILL command will be processed properly > > by > > > > all > > > > > > kinds of SQL queries - SELECT/DML/DDL, non-transactional or > > > > > transactional, > > > > > > local queries and distributed queries. Will we be able to support > > all > > > > > these > > > > > > modes? > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > On Mon, Nov 19, 2018 at 6:37 PM Юрий < > jury.gerzhedow...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > Hi Igniters, > > > > > > > > > >
Re: New API for changing configuration of persistent caches
Ed, We may have API similar to “cache” and “getOrCreateCache”, or may not. It is up to us to decide. Similarity on it’s own is weak argument. Functionality and simplicity - this is what matters. Approach with cache configuration has three major issues 1) It exposes properties which user will not be able to change, so typical user actions would be: try to change property, fail as it is unsupported, go reading documentation. Approach with separate POJO is intuitive and self-documenting. 2) It has race condition between config read and config apply, so user do not know what exactly he changes, unless you change API to something like “restartCaches(Tuple...)”, which user will need to call in a loop. 3) And it is not suitable for non-Java platform, which is a showstopper - all API should be available from all platforms unless it is proven to be impossible to implement. Vladimir. чт, 22 нояб. 2018 г. в 1:06, Eduard Shangareev : > Vovan, > > Would you argue that we should have the similar API in Java as > Ignite.cache(CacheConfiguration) or > Ignite.getOrCreateCache(CacheConfiguration)? > > With a proposed solution, every other API call would rely on it finally. > > I am interested in having such feature not arguing about API alternatives. > > We definitely should have the ability to change it via control.sh and Java > API. Everything else is optional from my point of view (at least on the > current stage). > > Moreover, your arguments are more about our format of CacheConfiguration > which couldn't be defined in other languages and clients. So, maybe we > should start a discussion about how we should change it in 3.0? > > > > > On Wed, Nov 21, 2018 at 7:45 PM Vladimir Ozerov > wrote: > > > Ed, > > > > Why do we want to operate on CacheConfiguration so desperately? Your > > example raises even more questions: > > 1) What to do with thin clients? > > 2) What to do with aforementioned race conditions, when cache could be > > changed concurrently? > > 3) Why such trivial operation from user perspective is only supported > from > > control.sh and not from the rest API (even Java client nodes will be > > affected - remember our plans to remove requirement to have cache classes > > on client nodes, which is yet to be implemented). > > > > Compare it to alternative API: > > > > 1) Native call from any language without limitations: > > > > > Ignite.changeCache(CacheConfigurationChange.create().setCacheMode(REPLICATED).setBackups(2)); > > > > 2) Call from control.sh in one line without race conditions with > concurrent > > cache changes: > > control.sh --cache --change cacheMode=REPLICATED backups=2 > > > > On Wed, Nov 21, 2018 at 7:32 PM Eduard Shangareev < > > eduard.shangar...@gmail.com> wrote: > > > > > Vovan, > > > > > > user already is able to get cache configuration as xml. > > > control.sh --cache list '.' --config > > > > > > So, user could update it and run: > > > control.sh --cache --restart -cfg=xml.path > > > > > > > > > On Wed, Nov 21, 2018 at 7:06 PM Vladimir Ozerov > > > wrote: > > > > > > > Ed, > > > > > > > > He can do that programmatically. But I meant another case - Java node > > > > creates a cache. Then .NET node wants to change it. Proposed API > cannot > > > > handle it. > > > > > > > > ср, 21 нояб. 2018 г. в 19:03, Eduard Shangareev < > > > eshangar...@gridgain.com > > > > >: > > > > > > > > > Vladimir, > > > > > > > > > > I didn't get how does .Net user start caches right now? XML and > > remote > > > > > node? Right? > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:55 PM Vladimir Ozerov < > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Ed, > > > > > > > > > > > > We are not Java product. We support 6 platforms at the moment. > Why > > do > > > > we > > > > > > implement a feature which can only be used in Java, when it is > very > > > > easy > > > > > to > > > > > > make it available from everywhere? > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:50, Eduard Shangareev < > > > > > eshangar...@gridgain.com > > > > > > >: > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > It would be Java API specific. > > > > > > > For a user, we would add a new command for console.sh which > would > > > > take > > > > > an > > > > > > > XML-file path as a parameter. > > > > > > > > > > > > > > We could add other possibilities: for example, with the builder > > > which > > > > > > would > > > > > > > finally call this Ignite.restartCaches method. But it's nice to > > > have, > > > > > > not a > > > > > > > mandatory one. > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:43 PM Vladimir Ozerov < > > > > voze...@gridgain.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Ed, > > > > > > > > > > > > > > > > Could you please demonstrate how .NET node or .NET will > change > > > > cache > > > > > > > > configuration with proposed API? Taking in count that XML is > > not > > > > > > > available > > > > > > > > in most cases, and custom Java classes from cache > conf
[MTCGA]: new failures in builds [2371886] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *New test failure in master CacheMvccPartitionedBackupsTest.testBackupsCoherenceWithLargeOperations https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-7557959062889759280&branch=%3Cdefault%3E&tab=testDetails *New test failure in master CacheMvccReplicatedBackupsTest.testBackupsCoherenceWithLargeOperations https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=6825219049988632083&branch=%3Cdefault%3E&tab=testDetails Changes may lead to failure were done by - gvvinblade https://ci.ignite.apache.org/viewModification.html?modId=839708 - andrey.mashenkov https://ci.ignite.apache.org/viewModification.html?modId=839694 - pudov.max https://ci.ignite.apache.org/viewModification.html?modId=839688 - ivandasch https://ci.ignite.apache.org/viewModification.html?modId=839748 - dmitrievanthony https://ci.ignite.apache.org/viewModification.html?modId=839684 - jokserfn https://ci.ignite.apache.org/viewModification.html?modId=839734 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 07:21:40 22-11-2018
[MTCGA]: new failures in builds [2371919] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. If your changes can lead to this failure(s): We're grateful that you were a volunteer to make the contribution to this project, but things change and you may no longer be able to finalize your contribution. Could you respond to this email and indicate if you wish to continue and fix test failures or step down and some committer may revert you commit. *New test failure in master IgniteLogicalRecoveryTest.testRecoveryOnCrushDuringCheckpointOnNodeStart https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-114025134572024&branch=%3Cdefault%3E&tab=testDetails Changes may lead to failure were done by - gvvinblade https://ci.ignite.apache.org/viewModification.html?modId=839708 - andrey.mashenkov https://ci.ignite.apache.org/viewModification.html?modId=839694 - pudov.max https://ci.ignite.apache.org/viewModification.html?modId=839688 - ivandasch https://ci.ignite.apache.org/viewModification.html?modId=839748 - dmitrievanthony https://ci.ignite.apache.org/viewModification.html?modId=839684 - jokserfn https://ci.ignite.apache.org/viewModification.html?modId=839734 - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 07:06:39 22-11-2018
Re: [IMPORTANT] Future of Binary Objects
We should definitely allow to change type of field/column to another
compatible type. The fact that we do not allow to change Int to Long is
pretty insane. However, there are cases when it's much more complicated.
How are we going to replace Int with a String, for example? I believe this
should require certain migration procedure anyway. How do other databases
handle that?
-Val
On Wed, Nov 21, 2018 at 4:32 PM Denis Magda wrote:
> Vladimir,
>
> Could you educate me a little bit, why the current format is bad for SQL
> and why another one is more suitable?
>
> Also, if we introduce the new format then why would we keep the binary one?
> Is the new format just a next version of the binary one.
>
> 2.3) Remove restrictions on changing field type
> > I do not know why we did that in the first place. This restriction
> prevents
> > type evolution and confuses users.
>
>
> That is a hot requirement shared by those who use Ignite SQL in production.
> +1.
>
> --
> Denis
>
> On Mon, Nov 19, 2018 at 11:05 PM Vladimir Ozerov
> wrote:
>
> > Igniters,
> >
> > It is very likely that Apache Ignite 3.0 will be released next year. So
> we
> > need to start thinking about major product improvements. I'd like to
> start
> > with binary objects.
> >
> > Currently they are one of the main limiting factors for the product. They
> > are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> > comparing to other vendors. They are slow - not suitable for SQL at all.
> >
> > I would like to ask all of you who worked with binary objects to share
> your
> > feedback and ideas, so that we understand how they should look like in AI
> > 3.0. This is a brain storm - let's accumulate ideas first and minimize
> > critics. Then we will work on ideas in separate topics.
> >
> > 1) Historical background
> >
> > BO were implemented around 2014 (Apache Ignite 1.5) when we started
> working
> > on .NET and CPP clients. During design we had several ideas in mind:
> > - ability to read object fields in O(1) without deserialization
> > - interoperabillty between Java, .NET and CPP.
> >
> > Since then a number of other concepts were mixed to the cocktail:
> > - Affinity key fields
> > - Strict typing for existing fields (aka metadata)
> > - Binary Object as storage format
> >
> > 2) My proposals
> >
> > 2.1) Introduce "Data Row Format" interface
> > Binary Objects are terrible candidates for storage. Too fat, too slow.
> > Efficient storage typically has <10 bytes overhead per row (no metadata,
> no
> > length, no hash code, etc), allow supper-fast field access, support
> > different string formats (ASCII, UTF-8, etc), support different temporal
> > types (date, time, timestamp, timestamp with timezone, etc), and store
> > these types as efficiently as possible.
> >
> > What we need is to introduce an interface which will convert a pair of
> > key-value objects into a row. This row will be used to store data and to
> > get fields from it. Care about memory consumption, need SQL and strict
> > schema - use one format. Need flexibility and prefer key-value access -
> use
> > another format which will store binary objects unchanged (current
> > behavior).
> >
> > interface DataRowFormat {
> > DataRow create(Object key, Object value); // primitives or binary
> > objects
> > DataRowMetadata metadata();
> > }
> >
> > 2.2) Remove affinity field from metadata
> > Affinity rules are governed by cache, not type. We should remove
> > "affintiyFieldName" from metadata.
> >
> > 2.3) Remove restrictions on changing field type
> > I do not know why we did that in the first place. This restriction
> prevents
> > type evolution and confuses users.
> >
> > 2.4) Use bitmaps for "null" and default values and for fixed-length
> fields,
> > put fixed-length fields before variable-length.
> > Motivation: to save space.
> >
> > What else? Please share your ideas.
> >
> > Vladimir.
> >
>
Re: [IMPORTANT] Future of Binary Objects
Vladimir,
Could you educate me a little bit, why the current format is bad for SQL
and why another one is more suitable?
Also, if we introduce the new format then why would we keep the binary one?
Is the new format just a next version of the binary one.
2.3) Remove restrictions on changing field type
> I do not know why we did that in the first place. This restriction prevents
> type evolution and confuses users.
That is a hot requirement shared by those who use Ignite SQL in production.
+1.
--
Denis
On Mon, Nov 19, 2018 at 11:05 PM Vladimir Ozerov
wrote:
> Igniters,
>
> It is very likely that Apache Ignite 3.0 will be released next year. So we
> need to start thinking about major product improvements. I'd like to start
> with binary objects.
>
> Currently they are one of the main limiting factors for the product. They
> are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> comparing to other vendors. They are slow - not suitable for SQL at all.
>
> I would like to ask all of you who worked with binary objects to share your
> feedback and ideas, so that we understand how they should look like in AI
> 3.0. This is a brain storm - let's accumulate ideas first and minimize
> critics. Then we will work on ideas in separate topics.
>
> 1) Historical background
>
> BO were implemented around 2014 (Apache Ignite 1.5) when we started working
> on .NET and CPP clients. During design we had several ideas in mind:
> - ability to read object fields in O(1) without deserialization
> - interoperabillty between Java, .NET and CPP.
>
> Since then a number of other concepts were mixed to the cocktail:
> - Affinity key fields
> - Strict typing for existing fields (aka metadata)
> - Binary Object as storage format
>
> 2) My proposals
>
> 2.1) Introduce "Data Row Format" interface
> Binary Objects are terrible candidates for storage. Too fat, too slow.
> Efficient storage typically has <10 bytes overhead per row (no metadata, no
> length, no hash code, etc), allow supper-fast field access, support
> different string formats (ASCII, UTF-8, etc), support different temporal
> types (date, time, timestamp, timestamp with timezone, etc), and store
> these types as efficiently as possible.
>
> What we need is to introduce an interface which will convert a pair of
> key-value objects into a row. This row will be used to store data and to
> get fields from it. Care about memory consumption, need SQL and strict
> schema - use one format. Need flexibility and prefer key-value access - use
> another format which will store binary objects unchanged (current
> behavior).
>
> interface DataRowFormat {
> DataRow create(Object key, Object value); // primitives or binary
> objects
> DataRowMetadata metadata();
> }
>
> 2.2) Remove affinity field from metadata
> Affinity rules are governed by cache, not type. We should remove
> "affintiyFieldName" from metadata.
>
> 2.3) Remove restrictions on changing field type
> I do not know why we did that in the first place. This restriction prevents
> type evolution and confuses users.
>
> 2.4) Use bitmaps for "null" and default values and for fixed-length fields,
> put fixed-length fields before variable-length.
> Motivation: to save space.
>
> What else? Please share your ideas.
>
> Vladimir.
>
Re: proposed realization KILL QUERY command
Vladimir, All of the alternatives are reminiscent of mathematical operations. Don't look like a SQL command. What if we use a SQL approach introducing named parameters: KILL QUERY query_id=10 [AND node_id=5] -- Denis On Wed, Nov 21, 2018 at 4:11 AM Vladimir Ozerov wrote: > Denis, > > Space is bad candidate because it is a whitespace. Without whitespaces we > can have syntax without quotes at all. Any non-whitespace delimiter will > work, though: > > KILL QUERY 45.1 > KILL QUERY 45-1 > KILL QUERY 45:1 > > On Wed, Nov 21, 2018 at 3:06 PM Юрий wrote: > > > Denis, > > > > Let's consider parameter of KILL QUERY just a string with some query id, > > without any meaning for user. User just need to get the id and pass as > > parameter to KILL QUERY command. > > > > Even if query is distributed it have single query id from user > perspective > > and will killed on all nodes. User just need to known one global query > id. > > > > How it can works. > > 1)SELECT * from running_queries > > result is > > query_id | node_id > > | sql | schema_name | connection_id | duration > > 123.33 | e0a69cb8-a1a8-45f6-b84d-ead367a0 | SELECT ... | ... > > | 22 | 23456 > > 333.31 | aaa6acb8-a4a5-42f6-f842-ead111b00020 | UPDATE... | ... > > | 321| 346 > > 2) KILL QUERY '123.33' > > > > So, user need select query_id from running_queries view and use it for > KILL > > QUERY command. > > > > I hope it became clearer. > > > > > > > > ср, 21 нояб. 2018 г. в 02:11, Denis Magda : > > > > > Folks, > > > > > > The decimal syntax is really odd - KILL QUERY > > > '[node_order].[query_counter]' > > > > > > Confusing, let's use a space to separate parameters. > > > > > > Also, what if I want to halt a specific query with certain ID? Don't > know > > > the node number, just know that the query is distributed and runs > across > > > several machines. Sounds like the syntax still should consider > > > [node_order/id] as an optional parameter. > > > > > > Probably, if you explain to me how an end user will use this command > from > > > the very beginning (how do I look for a query id and node id, etc) then > > the > > > things get clearer. > > > > > > -- > > > Denis > > > > > > On Tue, Nov 20, 2018 at 1:03 AM Юрий > > wrote: > > > > > > > Hi Vladimir, > > > > > > > > Thanks for your suggestion to use MANAGEMENT_POOL for processing > > > > cancellation requests. > > > > > > > > About your questions. > > > > 1) I'm going to implements SQL view to provide list of running > queries. > > > The > > > > SQL VIEW has been a little bit discussed earlier. Proposed name is > > > > *running_queries* with following columns: query_id, node_id, sql, > > > > schema_name, connection_id, duration. Currently most of the > information > > > can > > > > be retrieved through cache API, however it doesn't matter, any case > we > > > > need to expose SQL VIEW. Seem's you are right - the part should be > > > > implemented firstly. > > > > 2) Fully agree that we need to support all kind of SQL queries > > > > (SLECT/DML/DDL, transactional, non transnational, local, > distributed). > > I > > > > definitely sure that it will possible for all of above, however I'm > not > > > > sure about DDL - need to investigate it deeper. Also need to > understand > > > > that canceled DML operation can lead to partially updated data for > non > > > > transational caches. > > > > > > > > > > > > > > > > пн, 19 нояб. 2018 г. в 19:17, Vladimir Ozerov >: > > > > > > > > > Hi Yuriy, > > > > > > > > > > I think we can use MANAGEMENT_POOL for this. It is already used for > > > some > > > > > internal Ignite tasks, and it appears to be a good candidate to > > process > > > > > cancel requests. > > > > > > > > > > But there are several things which are not clear enough for me at > the > > > > > moment: > > > > > 1) How user is going to get the list of running queries in the > first > > > > place? > > > > > Do we already have any SQL commands/views to get this information? > > > > > 2) We need to ensure that KILL command will be processed properly > by > > > all > > > > > kinds of SQL queries - SELECT/DML/DDL, non-transactional or > > > > transactional, > > > > > local queries and distributed queries. Will we be able to support > all > > > > these > > > > > modes? > > > > > > > > > > Vladimir. > > > > > > > > > > On Mon, Nov 19, 2018 at 6:37 PM Юрий > > > > wrote: > > > > > > > > > > > Hi Igniters, > > > > > > > > > > > > Earlier we agreed about syntax KILL QUERY > > > > '[node_order].[query_counter]', > > > > > > e.g. KILL QUERY '25.123' for single query or KILL QUERY '25.*' > for > > > all > > > > > > queries on the node. Which is part of IEP-29 > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring > > > > > > > > > > > > > . > > > > > > > > > > > > Now I want to discuss internal realizati
Re: New API for changing configuration of persistent caches
Vovan, Would you argue that we should have the similar API in Java as Ignite.cache(CacheConfiguration) or Ignite.getOrCreateCache(CacheConfiguration)? With a proposed solution, every other API call would rely on it finally. I am interested in having such feature not arguing about API alternatives. We definitely should have the ability to change it via control.sh and Java API. Everything else is optional from my point of view (at least on the current stage). Moreover, your arguments are more about our format of CacheConfiguration which couldn't be defined in other languages and clients. So, maybe we should start a discussion about how we should change it in 3.0? On Wed, Nov 21, 2018 at 7:45 PM Vladimir Ozerov wrote: > Ed, > > Why do we want to operate on CacheConfiguration so desperately? Your > example raises even more questions: > 1) What to do with thin clients? > 2) What to do with aforementioned race conditions, when cache could be > changed concurrently? > 3) Why such trivial operation from user perspective is only supported from > control.sh and not from the rest API (even Java client nodes will be > affected - remember our plans to remove requirement to have cache classes > on client nodes, which is yet to be implemented). > > Compare it to alternative API: > > 1) Native call from any language without limitations: > > Ignite.changeCache(CacheConfigurationChange.create().setCacheMode(REPLICATED).setBackups(2)); > > 2) Call from control.sh in one line without race conditions with concurrent > cache changes: > control.sh --cache --change cacheMode=REPLICATED backups=2 > > On Wed, Nov 21, 2018 at 7:32 PM Eduard Shangareev < > eduard.shangar...@gmail.com> wrote: > > > Vovan, > > > > user already is able to get cache configuration as xml. > > control.sh --cache list '.' --config > > > > So, user could update it and run: > > control.sh --cache --restart -cfg=xml.path > > > > > > On Wed, Nov 21, 2018 at 7:06 PM Vladimir Ozerov > > wrote: > > > > > Ed, > > > > > > He can do that programmatically. But I meant another case - Java node > > > creates a cache. Then .NET node wants to change it. Proposed API cannot > > > handle it. > > > > > > ср, 21 нояб. 2018 г. в 19:03, Eduard Shangareev < > > eshangar...@gridgain.com > > > >: > > > > > > > Vladimir, > > > > > > > > I didn't get how does .Net user start caches right now? XML and > remote > > > > node? Right? > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:55 PM Vladimir Ozerov < > voze...@gridgain.com> > > > > wrote: > > > > > > > > > Ed, > > > > > > > > > > We are not Java product. We support 6 platforms at the moment. Why > do > > > we > > > > > implement a feature which can only be used in Java, when it is very > > > easy > > > > to > > > > > make it available from everywhere? > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:50, Eduard Shangareev < > > > > eshangar...@gridgain.com > > > > > >: > > > > > > > > > > > Vladimir, > > > > > > > > > > > > It would be Java API specific. > > > > > > For a user, we would add a new command for console.sh which would > > > take > > > > an > > > > > > XML-file path as a parameter. > > > > > > > > > > > > We could add other possibilities: for example, with the builder > > which > > > > > would > > > > > > finally call this Ignite.restartCaches method. But it's nice to > > have, > > > > > not a > > > > > > mandatory one. > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:43 PM Vladimir Ozerov < > > > voze...@gridgain.com> > > > > > > wrote: > > > > > > > > > > > > > Ed, > > > > > > > > > > > > > > Could you please demonstrate how .NET node or .NET will change > > > cache > > > > > > > configuration with proposed API? Taking in count that XML is > not > > > > > > available > > > > > > > in most cases, and custom Java classes from cache configuration > > are > > > > > > > available only on server nodes and only from Java. > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:36, Eduard Shangareev < > > > > > > > eduard.shangar...@gmail.com > > > > > > > >: > > > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > > > I don't see any difference here. > > > > > > > > > > > > > > > > The same possibilities would be available as with normal > cache > > > > start: > > > > > > > > -XML; > > > > > > > > -remote node. > > > > > > > > > > > > > > > > >3) Avoid race condition when configuration changes between > > > > > > configuration > > > > > > > > read and method call (what could lead to a number of strange > > > > > effects). > > > > > > > > > > > > > > > > Well, we could add *old* configuration parameter for CAS-like > > > > > semantic. > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov < > > > > > voze...@gridgain.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Ed, > > > > > > > > > > > > > > > > > > Caches in .NET could be started programmatically, from XML > > > which > > > > > .NET > > > > > > > API > > > > > > > > > has no access to, or dynamically from remote n
[jira] [Created] (IGNITE-10372) [TC Bot] Optimize master trends page
Dmitriy Pavlov created IGNITE-10372: --- Summary: [TC Bot] Optimize master trends page Key: IGNITE-10372 URL: https://issues.apache.org/jira/browse/IGNITE-10372 Project: Ignite Issue Type: Task Reporter: Dmitriy Pavlov Assignee: Dmitriy Pavlov Now loading of the page may take up to several minutes. It may be optimized - by avoiding tests list obtaining from composites (instead we may load chain of sub-builds) - by creating a cache of loaded builds in memory (e.g. guava soft values for the main method) - by excluding of usage of deprecated REST caching API approach -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #5467: Ignite 2.5.1 p161
GitHub user antonovsergey93 opened a pull request: https://github.com/apache/ignite/pull/5467 Ignite 2.5.1 p161 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-2.5.1-p161 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5467.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5467 commit 47d299c4961143f27d1f8aaa0d8c8e26b8ade7ed Author: Evgeny Stanilovskiy Date: 2018-07-23T08:56:21Z IGNITE-8892 Fixed OOME when scan query is used for a big partition - Fixes #4391. Signed-off-by: Alexey Goncharuk (cherry picked from commit a164296) commit a6a09865f776453ed88c093f4a58852ecb7efb0e Author: Andrey V. Mashenkov Date: 2018-07-23T09:39:17Z Merge remote-tracking branch 'origin/ignite-2.5.1-master' into ignite-2.5.1-master commit e2695cd960b7f0cbbba545ef60645ea964f3d7fa Author: Dmitriy Govorukhin Date: 2018-07-23T08:25:49Z IGNITE-9042 Fixed partial tranasaction state wheh transaction is timed out - Fixes #4397. Signed-off-by: Alexey Goncharuk (cherry picked from commit 33f485a) commit 702c31859251738b187e8db2a8155264f39b8b52 Author: Ivan Daschinskiy Date: 2018-07-23T12:29:08Z IGNITE-8820 Fix rollback logic when tx is initiated from client. - Fixes #4399. Signed-off-by: Alexey Goncharuk (cherry picked from commit 5794eb0) commit 32777e77797f4cf3260d42d4ca408336247d4e55 Author: Dmitriy Govorukhin Date: 2018-07-23T15:01:37Z IGNITE-9049 Fixed write of SWITCH_SEGMENT_RECORD at the end of a segment file - Fixes #4401. Signed-off-by: Alexey Goncharuk (cherry picked from commit 713a428) commit bbf8f83afe3b07e807912d9bab78873edf698d4d Author: Andrey V. Mashenkov Date: 2018-07-24T14:38:34Z IGNITE-8892 Fixed CacheQueryFuture usage in DataStructures processor - Fixes #4415. Signed-off-by: Alexey Goncharuk (cherry picked from commit 281a400) commit e0404913088b5cd97cd79de1b62e6aaa4c00b5d2 Author: Andrey V. Mashenkov Date: 2018-07-24T15:24:57Z Merge remote-tracking branch 'origin/ignite-2.5.1-master' into ignite-2.5.1-master commit 86c52700a007368a351ac3595a8e38bb02923080 Author: Sergey Chugunov Date: 2018-07-25T13:26:12Z IGNITE-9040 StopNodeFailureHandler is not able to stop node correctly on node segmentation Signed-off-by: Andrey Gura (cherry-picked from commit#469aaba59c0539507972f4725642b2f2f81c08a0) commit e1c3cc9d30baeaff4e653751775ab39a347b753b Author: Pavel Kovalenko Date: 2018-07-24T14:48:57Z IGNITE-8791 Fixed missed update counter in WAL data record for backup transaction - Fixes #4264. Signed-off-by: Alexey Goncharuk (cherry picked from commit 13e2a31) commit 2ea5b376d895fe41cc882c4383b5d3d2793ba684 Author: devozerov Date: 2018-07-31T09:53:51Z IGNITE-9114: SQL: fail query after some timeout if it cannot be mapped to topology. This closes #4453. commit 5b230b7015148e483136c229c9e6081bb9c68023 Author: Denis Mekhanikov Date: 2018-04-20T15:41:06Z IGNITE-8134 Subscribe to system cache events on nodes outside BLT Signed-off-by: Andrey Gura (cherry picked from commit c82277e) commit 2528435ec946c479f6bf4eb7eaeabf092a0536a3 Author: devozerov Date: 2018-07-31T14:15:49Z IGNITE-9114: SQL: use query time in retry timeout calculation. This closes #4460. commit 98d0ccc729e647a4ad8876e2d29423904e6fa6ca Author: devozerov Date: 2018-07-31T14:18:49Z Merge remote-tracking branch 'upstream/ignite-2.5.1-master' into ignite-2.5.1-master commit c78a7f215699f65983edfcc0c9014d9e3108f381 Author: Anton Kalashnikov Date: 2018-08-01T09:23:03Z IGNITE-8757 idle_verify utility doesn't show both update counter and hash conflicts (cherry picked from commit 9131e4d) commit 66369583096b04d71c3f82e1ebfd1922ae6b0b6a Author: Anton Kalashnikov Date: 2018-07-31T13:13:27Z IGNITE-8973 Calculate partition hash and print into standard output Signed-off-by: Andrey Gura (cherry picked from commit 8309cef) commit e92788d392d4c60f76c4fd0f38c52669f4e3e772 Author: Evgeny Stanilovskiy Date: 2018-08-01T11:05:58Z IGNITE-8939 Catch and proper propagate transaction string reprsentation - Fixes #4454. Signed-off-by: Dmitriy Pavlov (cherry picked from commit 458480c5b0520aa8e28935361a37ab49e1e65ff6) commit 60155cb55ca35790da7f80ae167dd7fb3f3793fd Author: Andrey V. Mashenkov Date: 2018-08-01T10:23:32Z IGNITE-9154: Fixed conflict version passed to events for ATOMIC cache. This closes #4463. (cherry picked from commit 3ea3a56) commit 15d72fc022f64939fee336438256460024ca59c0 Author: Andrey V. Mashenkov Date: 2018-08-01T12:58:22Z Merge remote-tracki
[GitHub] ignite pull request #5466: Ignite 1793 test mass run.
GitHub user xtern reopened a pull request: https://github.com/apache/ignite/pull/5466 Ignite 1793 test mass run. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xtern/ignite IGNITE-1793 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5466.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5466 commit b994a2f0c817763007472cf24caca142af82d882 Author: pereslegin-pa Date: 2018-11-21T16:47:08Z TC test mass run. commit 8664d101f82f818152bc62212fc60ecd35134836 Author: pereslegin-pa Date: 2018-11-21T20:12:01Z reduced run count ---
[GitHub] ignite pull request #5466: Ignite 1793 test mass run.
Github user xtern closed the pull request at: https://github.com/apache/ignite/pull/5466 ---
Re: MVCC test coverage.
Hi Andrey, Thank you for bringing this question to the list. I already reviewed this PR and it looks good to me. But I would like to hear more opinions from other community members regarding the whole approach. One important detail - we are going to create new suites as a child classes of existing suites with irrelevant tests excluded manually. This way if a new test is added to existing cache suite, it will be automatically added to TC suite as well, and we will see potential MVCC issues in a nightly build. This is critical thing to keep MVCC mode on par with “classical” transactions. I am not 100% happy with the fact that we will know about new failures only after problematic commit is pushed. But I do not see how to improve it without extending Run All time for another 30 hours. This will do more harm than good. So proposed solution looks like a good pros-cons balance at the moment. Vladimir. ср, 21 нояб. 2018 г. в 17:59, Andrey Mashenkov : > Hi Igniters, > > As you may already know, MVCC transaction feature will be available in > upcoming Ignite-2.7. > However, MVCC Tx feature is released for beta testing and has many > limitations and we a going > to improve stability and integrations with other features in future > releases. > > We can reuse existed transactional cache tests and run them in Mvcc mode to > get much better test coverage with small looses. > Here is a ticket for this IGNITE-10001 [1]. > > This mean we will have twice more "Cache Tests" and get TC runs some > longer. > To reduce new Mvcc cache suites impact and save TC time we are going to > 1. Include new tests to nightly suite only, this will allow us to put our > ears to the ground. > 2. Exclude non-tx tests and non-relevant tx cases and aggressively mute > tests for unsupported features integrations. > > I've implement a PR to one of tasks [2] as an example how it can be done. > > Technical details: > 1. Introduced a new FORCE_MVCC flag and created a child "Mvcc Cache 2" > suite for "Cache 2" test suite with FORCE_MVCC flag on. > 2. Implemented a hook that change TRANSACTIONAL cache atomicity mode to > TRANSACTIONAL_SNAPSHOT if FORCE_MVCC flag turned on. > This allow us to check MVCC mode without creating new test classes and > minimal intervention in existed tests code. > 3. All irrelevant tests marked as ignored in Mvcc suite. > 4. Known unsupported cases are muted. New failures muted as well > (corresponding tickets were created). > > > Any pros\cons? > Can someone please review a PR? > > [1] https://issues.apache.org/jira/browse/IGNITE-10001 > [2] https://issues.apache.org/jira/browse/IGNITE-10002 > -- > Best regards, > Andrey V. Mashenkov >
[GitHub] ignite pull request #4835: IGNITE-9695 WAL disabling prohibition in WalState...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4835 ---
Re: Mass TC RunAll: 8th iteration
Hi Igniters, For a long time, I did not collect MTCGA reports because now we have a very convenient statistics page for the Bot ( https://goo.gl/4Tj1mo ). But today I've decided to check our progress since August. As always you can find full test report from 30 latest runs here https://cwiki.apache.org/confluence/pages/viewpageattachments.action?pageId=73631266 The current snapshot of the latest 30 run-all shows 1158 test failures, 29 suites critical failures (timeout/crash/exit code/OOME). Suites failures have almost the same rate, while tests failures were doubled. Sincerely, Dmitriy Pavlov пн, 6 авг. 2018 г. в 15:28, Dmitriy Pavlov : > Hi Igniters, > > I’m glad there is positive dynamic during last 3 weeks related to test > failures. Latest weekend mass run-all shows tests 427 failures, suites 24 > critical failures (timeout/crash/exitcode/OOME). 3 weeks ago that metrics > were as follows: tests 512 and suites 41. > > Please see below for a link to full report > > > https://cwiki.apache.org/confluence/pages/viewpageattachments.action?pageId=73631266 > > Thanks very much to Eduard Shangareev, Vitaliy Biryukov, Semen Boikov, > Pavel Kovalenko, Maxim Muzafarov, Alexey Stelmak, Alexey Goncharuk, Anton > Kalashnikov, Ivan Daschinskiy, Sergey Chugunov, Aleksey Plekhanov, Anton > Vinogradov, Mikhail Cherkasov, Amelchev Nikita, Denis Garus, Ivan Fedotov, > Ryabov Dmitrii for their contributions, all other contributors, who was > involved into fixing tests, and to everyone that helped with reviewing and > merging to master. > > Sincerely, > > Dmitriy Pavlov > > > пт, 3 авг. 2018 г. в 13:03, Dmitriy Pavlov : > >> Hi Igniters, >> >> I'm going to trigger mass run all to be executed during weekend. I hope >> you don't mind. >> >> I will trigger build 17:00 UTC, 20:00 MSK, 10:00 PST. Should you have >> urgent PR to be checked, feel free to reorder it to top. >> >> Sincerely, >> Dmitriy Pavlov >> >
[GitHub] ignite pull request #5458: IGNITE-10339 Fix connection to cluster failed whe...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5458 ---
[jira] [Created] (IGNITE-10371) [ML] Add multiple metrics calculation fo Binary Classification Evaluation process
Aleksey Zinoviev created IGNITE-10371: - Summary: [ML] Add multiple metrics calculation fo Binary Classification Evaluation process Key: IGNITE-10371 URL: https://issues.apache.org/jira/browse/IGNITE-10371 Project: Ignite Issue Type: New Feature Components: ml Affects Versions: 2.8 Reporter: Aleksey Zinoviev Assignee: Aleksey Zinoviev Fix For: 2.8 Add ability to get map of metrics to evaluate binary classification. Try to implement: All implemented metrics should be calculated for one iteration cycle along the data Naive implementation: compose all passed metrics and calculate them separatly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #77: IGNITE-10336: Build chain collection refactored to avoid missing builds
asfgit closed pull request #77: IGNITE-10336: Build chain collection refactored to avoid missing builds URL: https://github.com/apache/ignite-teamcity-bot/pull/77 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: [IMPORTANT] Future of Binary Objects
Hi,
Vladimir, Ilya,
What about variable length fields? How do you suggest to store offsets in
footer or header?
For large objects, headers will allow to retrive field faster and detect
null immediately, but we have to reserve place for all var-len fields
offset and update header after serialization.
however, footers looks more compact (we can omit nulls) and allow us to use
stream concept during serialization.
Have I miss smth?
On Wed, Nov 21, 2018 at 7:18 PM Ilya Kasnacheev
wrote:
> Hello!
>
> I would like to propose the following changes:
>
> - Let's allow multiple BinaryType's per Class. Make typeId = cksum(list of
> class types + fields) as opposed of cksum(class name) as we have it
> currently. Note that we only have to compute that once per class loaded in
> JVM.
> - BinaryType has a list of fixed length fields (numbers, datetimes, flags)
> and list of variable length fields. We can put all fixed length fields at
> start of BinaryObject so that we can access them by offset as per typeId.
> - Likewise we don't need to encode field id in BinaryObject anymore, save 4
> bytes per field. We already know their order from BinaryType.
> - This means when you ALTER TABLE we add a BinaryType to existing Class (or
> pseudo-Class type name) and we can use it for new data, and eventually
> update existing data to have this field.
> - On top of BinaryType's we can have checks that run them against SQL table
> columns list to see if there are any mismatches.
>
> To Illustrate, previously we had it like:
> [ Type id | String field id | String field value | Long field id | Long
> field value | Datetime field id | Datetime field value ]
> But now it will be
> [ Type id | Long field value | Datetime field value | String field value ]
> ^--^ can be accessed by offset
>
> Regards,
> Ilya.
>
> --
> Ilya Kasnacheev
>
>
> вт, 20 нояб. 2018 г. в 10:05, Vladimir Ozerov :
>
> > Igniters,
> >
> > It is very likely that Apache Ignite 3.0 will be released next year. So
> we
> > need to start thinking about major product improvements. I'd like to
> start
> > with binary objects.
> >
> > Currently they are one of the main limiting factors for the product. They
> > are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> > comparing to other vendors. They are slow - not suitable for SQL at all.
> >
> > I would like to ask all of you who worked with binary objects to share
> your
> > feedback and ideas, so that we understand how they should look like in AI
> > 3.0. This is a brain storm - let's accumulate ideas first and minimize
> > critics. Then we will work on ideas in separate topics.
> >
> > 1) Historical background
> >
> > BO were implemented around 2014 (Apache Ignite 1.5) when we started
> working
> > on .NET and CPP clients. During design we had several ideas in mind:
> > - ability to read object fields in O(1) without deserialization
> > - interoperabillty between Java, .NET and CPP.
> >
> > Since then a number of other concepts were mixed to the cocktail:
> > - Affinity key fields
> > - Strict typing for existing fields (aka metadata)
> > - Binary Object as storage format
> >
> > 2) My proposals
> >
> > 2.1) Introduce "Data Row Format" interface
> > Binary Objects are terrible candidates for storage. Too fat, too slow.
> > Efficient storage typically has <10 bytes overhead per row (no metadata,
> no
> > length, no hash code, etc), allow supper-fast field access, support
> > different string formats (ASCII, UTF-8, etc), support different temporal
> > types (date, time, timestamp, timestamp with timezone, etc), and store
> > these types as efficiently as possible.
> >
> > What we need is to introduce an interface which will convert a pair of
> > key-value objects into a row. This row will be used to store data and to
> > get fields from it. Care about memory consumption, need SQL and strict
> > schema - use one format. Need flexibility and prefer key-value access -
> use
> > another format which will store binary objects unchanged (current
> > behavior).
> >
> > interface DataRowFormat {
> > DataRow create(Object key, Object value); // primitives or binary
> > objects
> > DataRowMetadata metadata();
> > }
> >
> > 2.2) Remove affinity field from metadata
> > Affinity rules are governed by cache, not type. We should remove
> > "affintiyFieldName" from metadata.
> >
> > 2.3) Remove restrictions on changing field type
> > I do not know why we did that in the first place. This restriction
> prevents
> > type evolution and confuses users.
> >
> > 2.4) Use bitmaps for "null" and default values and for fixed-length
> fields,
> > put fixed-length fields before variable-length.
> > Motivation: to save space.
> >
> > What else? Please share your ideas.
> >
> > Vladimir.
> >
>
--
Best regards,
Andrey V. Mashenkov
[GitHub] ignite pull request #5466: Ignite 1793 test mass run.
GitHub user xtern opened a pull request: https://github.com/apache/ignite/pull/5466 Ignite 1793 test mass run. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xtern/ignite IGNITE-1793 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5466.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5466 commit b994a2f0c817763007472cf24caca142af82d882 Author: pereslegin-pa Date: 2018-11-21T16:47:08Z TC test mass run. ---
Re: New API for changing configuration of persistent caches
Ed, Why do we want to operate on CacheConfiguration so desperately? Your example raises even more questions: 1) What to do with thin clients? 2) What to do with aforementioned race conditions, when cache could be changed concurrently? 3) Why such trivial operation from user perspective is only supported from control.sh and not from the rest API (even Java client nodes will be affected - remember our plans to remove requirement to have cache classes on client nodes, which is yet to be implemented). Compare it to alternative API: 1) Native call from any language without limitations: Ignite.changeCache(CacheConfigurationChange.create().setCacheMode(REPLICATED).setBackups(2)); 2) Call from control.sh in one line without race conditions with concurrent cache changes: control.sh --cache --change cacheMode=REPLICATED backups=2 On Wed, Nov 21, 2018 at 7:32 PM Eduard Shangareev < eduard.shangar...@gmail.com> wrote: > Vovan, > > user already is able to get cache configuration as xml. > control.sh --cache list '.' --config > > So, user could update it and run: > control.sh --cache --restart -cfg=xml.path > > > On Wed, Nov 21, 2018 at 7:06 PM Vladimir Ozerov > wrote: > > > Ed, > > > > He can do that programmatically. But I meant another case - Java node > > creates a cache. Then .NET node wants to change it. Proposed API cannot > > handle it. > > > > ср, 21 нояб. 2018 г. в 19:03, Eduard Shangareev < > eshangar...@gridgain.com > > >: > > > > > Vladimir, > > > > > > I didn't get how does .Net user start caches right now? XML and remote > > > node? Right? > > > > > > > > > On Wed, Nov 21, 2018 at 6:55 PM Vladimir Ozerov > > > wrote: > > > > > > > Ed, > > > > > > > > We are not Java product. We support 6 platforms at the moment. Why do > > we > > > > implement a feature which can only be used in Java, when it is very > > easy > > > to > > > > make it available from everywhere? > > > > > > > > ср, 21 нояб. 2018 г. в 18:50, Eduard Shangareev < > > > eshangar...@gridgain.com > > > > >: > > > > > > > > > Vladimir, > > > > > > > > > > It would be Java API specific. > > > > > For a user, we would add a new command for console.sh which would > > take > > > an > > > > > XML-file path as a parameter. > > > > > > > > > > We could add other possibilities: for example, with the builder > which > > > > would > > > > > finally call this Ignite.restartCaches method. But it's nice to > have, > > > > not a > > > > > mandatory one. > > > > > > > > > > On Wed, Nov 21, 2018 at 6:43 PM Vladimir Ozerov < > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Ed, > > > > > > > > > > > > Could you please demonstrate how .NET node or .NET will change > > cache > > > > > > configuration with proposed API? Taking in count that XML is not > > > > > available > > > > > > in most cases, and custom Java classes from cache configuration > are > > > > > > available only on server nodes and only from Java. > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:36, Eduard Shangareev < > > > > > > eduard.shangar...@gmail.com > > > > > > >: > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > I don't see any difference here. > > > > > > > > > > > > > > The same possibilities would be available as with normal cache > > > start: > > > > > > > -XML; > > > > > > > -remote node. > > > > > > > > > > > > > > >3) Avoid race condition when configuration changes between > > > > > configuration > > > > > > > read and method call (what could lead to a number of strange > > > > effects). > > > > > > > > > > > > > > Well, we could add *old* configuration parameter for CAS-like > > > > semantic. > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov < > > > > voze...@gridgain.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Ed, > > > > > > > > > > > > > > > > Caches in .NET could be started programmatically, from XML > > which > > > > .NET > > > > > > API > > > > > > > > has no access to, or dynamically from remote nodes (eg Java > > > node). > > > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev < > > > > > > > > eduard.shangar...@gmail.com > > > > > > > > >: > > > > > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > > > > > How does .Net user start caches right now? > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov < > > > > > > voze...@gridgain.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Eduard, > > > > > > > > > > > > > > > > > > > > Simple != correct. Let’s consider a simple use case: user > > > want > > > > to > > > > > > > > change > > > > > > > > > > PARTITIONED -> REPLICATED from .NET, but do not some > > classes > > > > from > > > > > > > > > > CacheConfiguration. How do we solve this? > > > > > > > > > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > > > > > > > > > eduard.shangar...@gmail.com > > > > > > > > > > >: > > > > > > > > > > > > >
[GitHub] ignite pull request #5465: ignite-8765: Fixed forgotten import.
GitHub user pavel-kuznetsov opened a pull request: https://github.com/apache/ignite/pull/5465 ignite-8765: Fixed forgotten import. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-8765-fix-imports Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5465.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5465 commit f0b67386db6c8106b62e731230bddec4dab7f14a Author: Pavel Kuznetsov Date: 2018-11-21T16:38:52Z ignite-8765: Fixed forgotten import. ---
Re: New API for changing configuration of persistent caches
By the very first look, a new method can work absolutely the same with cache() & getOrCreateCache() - it will be easy to follow by users. Vladimir, could you give an example of API you proposing to implement? ср, 21 нояб. 2018 г. в 19:32, Eduard Shangareev : > Vovan, > > user already is able to get cache configuration as xml. > control.sh --cache list '.' --config > > So, user could update it and run: > control.sh --cache --restart -cfg=xml.path > > > On Wed, Nov 21, 2018 at 7:06 PM Vladimir Ozerov > wrote: > > > Ed, > > > > He can do that programmatically. But I meant another case - Java node > > creates a cache. Then .NET node wants to change it. Proposed API cannot > > handle it. > > > > ср, 21 нояб. 2018 г. в 19:03, Eduard Shangareev < > eshangar...@gridgain.com > > >: > > > > > Vladimir, > > > > > > I didn't get how does .Net user start caches right now? XML and remote > > > node? Right? > > > > > > > > > On Wed, Nov 21, 2018 at 6:55 PM Vladimir Ozerov > > > wrote: > > > > > > > Ed, > > > > > > > > We are not Java product. We support 6 platforms at the moment. Why do > > we > > > > implement a feature which can only be used in Java, when it is very > > easy > > > to > > > > make it available from everywhere? > > > > > > > > ср, 21 нояб. 2018 г. в 18:50, Eduard Shangareev < > > > eshangar...@gridgain.com > > > > >: > > > > > > > > > Vladimir, > > > > > > > > > > It would be Java API specific. > > > > > For a user, we would add a new command for console.sh which would > > take > > > an > > > > > XML-file path as a parameter. > > > > > > > > > > We could add other possibilities: for example, with the builder > which > > > > would > > > > > finally call this Ignite.restartCaches method. But it's nice to > have, > > > > not a > > > > > mandatory one. > > > > > > > > > > On Wed, Nov 21, 2018 at 6:43 PM Vladimir Ozerov < > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Ed, > > > > > > > > > > > > Could you please demonstrate how .NET node or .NET will change > > cache > > > > > > configuration with proposed API? Taking in count that XML is not > > > > > available > > > > > > in most cases, and custom Java classes from cache configuration > are > > > > > > available only on server nodes and only from Java. > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:36, Eduard Shangareev < > > > > > > eduard.shangar...@gmail.com > > > > > > >: > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > I don't see any difference here. > > > > > > > > > > > > > > The same possibilities would be available as with normal cache > > > start: > > > > > > > -XML; > > > > > > > -remote node. > > > > > > > > > > > > > > >3) Avoid race condition when configuration changes between > > > > > configuration > > > > > > > read and method call (what could lead to a number of strange > > > > effects). > > > > > > > > > > > > > > Well, we could add *old* configuration parameter for CAS-like > > > > semantic. > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov < > > > > voze...@gridgain.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Ed, > > > > > > > > > > > > > > > > Caches in .NET could be started programmatically, from XML > > which > > > > .NET > > > > > > API > > > > > > > > has no access to, or dynamically from remote nodes (eg Java > > > node). > > > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev < > > > > > > > > eduard.shangar...@gmail.com > > > > > > > > >: > > > > > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > > > > > How does .Net user start caches right now? > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov < > > > > > > voze...@gridgain.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Eduard, > > > > > > > > > > > > > > > > > > > > Simple != correct. Let’s consider a simple use case: user > > > want > > > > to > > > > > > > > change > > > > > > > > > > PARTITIONED -> REPLICATED from .NET, but do not some > > classes > > > > from > > > > > > > > > > CacheConfiguration. How do we solve this? > > > > > > > > > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > > > > > > > > > eduard.shangar...@gmail.com > > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > > > > > > > > > I propose not to change cache configuration in runtime > > but > > > > > > restart > > > > > > > > > cache > > > > > > > > > > > with the new compatible configuration on data which we > > have > > > > > > > > underfoot. > > > > > > > > > > > > > > > > > > > > > > What we could change: > > > > > > > > > > > -backup count; > > > > > > > > > > > -TRANSACTIONAL <-> ATOMIC; > > > > > > > > > > > -REPLICATED - PARTITIONED; > > > > > > > > > > > -other settings. > > > > > > > > > > > > > > > > > > > > > > So, yeah, it would be great to have a possibility to > > change > > > > > some > > > > >
Re: New API for changing configuration of persistent caches
Vovan, user already is able to get cache configuration as xml. control.sh --cache list '.' --config So, user could update it and run: control.sh --cache --restart -cfg=xml.path On Wed, Nov 21, 2018 at 7:06 PM Vladimir Ozerov wrote: > Ed, > > He can do that programmatically. But I meant another case - Java node > creates a cache. Then .NET node wants to change it. Proposed API cannot > handle it. > > ср, 21 нояб. 2018 г. в 19:03, Eduard Shangareev >: > > > Vladimir, > > > > I didn't get how does .Net user start caches right now? XML and remote > > node? Right? > > > > > > On Wed, Nov 21, 2018 at 6:55 PM Vladimir Ozerov > > wrote: > > > > > Ed, > > > > > > We are not Java product. We support 6 platforms at the moment. Why do > we > > > implement a feature which can only be used in Java, when it is very > easy > > to > > > make it available from everywhere? > > > > > > ср, 21 нояб. 2018 г. в 18:50, Eduard Shangareev < > > eshangar...@gridgain.com > > > >: > > > > > > > Vladimir, > > > > > > > > It would be Java API specific. > > > > For a user, we would add a new command for console.sh which would > take > > an > > > > XML-file path as a parameter. > > > > > > > > We could add other possibilities: for example, with the builder which > > > would > > > > finally call this Ignite.restartCaches method. But it's nice to have, > > > not a > > > > mandatory one. > > > > > > > > On Wed, Nov 21, 2018 at 6:43 PM Vladimir Ozerov < > voze...@gridgain.com> > > > > wrote: > > > > > > > > > Ed, > > > > > > > > > > Could you please demonstrate how .NET node or .NET will change > cache > > > > > configuration with proposed API? Taking in count that XML is not > > > > available > > > > > in most cases, and custom Java classes from cache configuration are > > > > > available only on server nodes and only from Java. > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:36, Eduard Shangareev < > > > > > eduard.shangar...@gmail.com > > > > > >: > > > > > > > > > > > Vladimir, > > > > > > > > > > > > I don't see any difference here. > > > > > > > > > > > > The same possibilities would be available as with normal cache > > start: > > > > > > -XML; > > > > > > -remote node. > > > > > > > > > > > > >3) Avoid race condition when configuration changes between > > > > configuration > > > > > > read and method call (what could lead to a number of strange > > > effects). > > > > > > > > > > > > Well, we could add *old* configuration parameter for CAS-like > > > semantic. > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov < > > > voze...@gridgain.com> > > > > > > wrote: > > > > > > > > > > > > > Ed, > > > > > > > > > > > > > > Caches in .NET could be started programmatically, from XML > which > > > .NET > > > > > API > > > > > > > has no access to, or dynamically from remote nodes (eg Java > > node). > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev < > > > > > > > eduard.shangar...@gmail.com > > > > > > > >: > > > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > > > How does .Net user start caches right now? > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov < > > > > > voze...@gridgain.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Eduard, > > > > > > > > > > > > > > > > > > Simple != correct. Let’s consider a simple use case: user > > want > > > to > > > > > > > change > > > > > > > > > PARTITIONED -> REPLICATED from .NET, but do not some > classes > > > from > > > > > > > > > CacheConfiguration. How do we solve this? > > > > > > > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > > > > > > > > eduard.shangar...@gmail.com > > > > > > > > > >: > > > > > > > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > > > > > > > I propose not to change cache configuration in runtime > but > > > > > restart > > > > > > > > cache > > > > > > > > > > with the new compatible configuration on data which we > have > > > > > > > underfoot. > > > > > > > > > > > > > > > > > > > > What we could change: > > > > > > > > > > -backup count; > > > > > > > > > > -TRANSACTIONAL <-> ATOMIC; > > > > > > > > > > -REPLICATED - PARTITIONED; > > > > > > > > > > -other settings. > > > > > > > > > > > > > > > > > > > > So, yeah, it would be great to have a possibility to > change > > > > some > > > > > > > > > properties > > > > > > > > > > in runtime. But right we don't any way to change anything > > > > except > > > > > > > > indexes > > > > > > > > > > and SQL fields. > > > > > > > > > > > > > > > > > > > > We already have all mechanism to do this. > > > > > > > > > > The main issue is to make it reliable and exclude cases > > when > > > we > > > > > > could > > > > > > > > > come > > > > > > > > > > to the unrecoverable state. > > > > > > > > > > > > > > > > > > > > So, I suggest keeping the solution as simple as possible. > > > > > > > > > > For indexes clashes and ClassNotFoundExcep
Re: [IMPORTANT] Future of Binary Objects
Hello!
I would like to propose the following changes:
- Let's allow multiple BinaryType's per Class. Make typeId = cksum(list of
class types + fields) as opposed of cksum(class name) as we have it
currently. Note that we only have to compute that once per class loaded in
JVM.
- BinaryType has a list of fixed length fields (numbers, datetimes, flags)
and list of variable length fields. We can put all fixed length fields at
start of BinaryObject so that we can access them by offset as per typeId.
- Likewise we don't need to encode field id in BinaryObject anymore, save 4
bytes per field. We already know their order from BinaryType.
- This means when you ALTER TABLE we add a BinaryType to existing Class (or
pseudo-Class type name) and we can use it for new data, and eventually
update existing data to have this field.
- On top of BinaryType's we can have checks that run them against SQL table
columns list to see if there are any mismatches.
To Illustrate, previously we had it like:
[ Type id | String field id | String field value | Long field id | Long
field value | Datetime field id | Datetime field value ]
But now it will be
[ Type id | Long field value | Datetime field value | String field value ]
^--^ can be accessed by offset
Regards,
Ilya.
--
Ilya Kasnacheev
вт, 20 нояб. 2018 г. в 10:05, Vladimir Ozerov :
> Igniters,
>
> It is very likely that Apache Ignite 3.0 will be released next year. So we
> need to start thinking about major product improvements. I'd like to start
> with binary objects.
>
> Currently they are one of the main limiting factors for the product. They
> are fat - 30+ bytes overhead on average, high TCO of Apache Ignite
> comparing to other vendors. They are slow - not suitable for SQL at all.
>
> I would like to ask all of you who worked with binary objects to share your
> feedback and ideas, so that we understand how they should look like in AI
> 3.0. This is a brain storm - let's accumulate ideas first and minimize
> critics. Then we will work on ideas in separate topics.
>
> 1) Historical background
>
> BO were implemented around 2014 (Apache Ignite 1.5) when we started working
> on .NET and CPP clients. During design we had several ideas in mind:
> - ability to read object fields in O(1) without deserialization
> - interoperabillty between Java, .NET and CPP.
>
> Since then a number of other concepts were mixed to the cocktail:
> - Affinity key fields
> - Strict typing for existing fields (aka metadata)
> - Binary Object as storage format
>
> 2) My proposals
>
> 2.1) Introduce "Data Row Format" interface
> Binary Objects are terrible candidates for storage. Too fat, too slow.
> Efficient storage typically has <10 bytes overhead per row (no metadata, no
> length, no hash code, etc), allow supper-fast field access, support
> different string formats (ASCII, UTF-8, etc), support different temporal
> types (date, time, timestamp, timestamp with timezone, etc), and store
> these types as efficiently as possible.
>
> What we need is to introduce an interface which will convert a pair of
> key-value objects into a row. This row will be used to store data and to
> get fields from it. Care about memory consumption, need SQL and strict
> schema - use one format. Need flexibility and prefer key-value access - use
> another format which will store binary objects unchanged (current
> behavior).
>
> interface DataRowFormat {
> DataRow create(Object key, Object value); // primitives or binary
> objects
> DataRowMetadata metadata();
> }
>
> 2.2) Remove affinity field from metadata
> Affinity rules are governed by cache, not type. We should remove
> "affintiyFieldName" from metadata.
>
> 2.3) Remove restrictions on changing field type
> I do not know why we did that in the first place. This restriction prevents
> type evolution and confuses users.
>
> 2.4) Use bitmaps for "null" and default values and for fixed-length fields,
> put fixed-length fields before variable-length.
> Motivation: to save space.
>
> What else? Please share your ideas.
>
> Vladimir.
>
Re: New API for changing configuration of persistent caches
Ed, He can do that programmatically. But I meant another case - Java node creates a cache. Then .NET node wants to change it. Proposed API cannot handle it. ср, 21 нояб. 2018 г. в 19:03, Eduard Shangareev : > Vladimir, > > I didn't get how does .Net user start caches right now? XML and remote > node? Right? > > > On Wed, Nov 21, 2018 at 6:55 PM Vladimir Ozerov > wrote: > > > Ed, > > > > We are not Java product. We support 6 platforms at the moment. Why do we > > implement a feature which can only be used in Java, when it is very easy > to > > make it available from everywhere? > > > > ср, 21 нояб. 2018 г. в 18:50, Eduard Shangareev < > eshangar...@gridgain.com > > >: > > > > > Vladimir, > > > > > > It would be Java API specific. > > > For a user, we would add a new command for console.sh which would take > an > > > XML-file path as a parameter. > > > > > > We could add other possibilities: for example, with the builder which > > would > > > finally call this Ignite.restartCaches method. But it's nice to have, > > not a > > > mandatory one. > > > > > > On Wed, Nov 21, 2018 at 6:43 PM Vladimir Ozerov > > > wrote: > > > > > > > Ed, > > > > > > > > Could you please demonstrate how .NET node or .NET will change cache > > > > configuration with proposed API? Taking in count that XML is not > > > available > > > > in most cases, and custom Java classes from cache configuration are > > > > available only on server nodes and only from Java. > > > > > > > > ср, 21 нояб. 2018 г. в 18:36, Eduard Shangareev < > > > > eduard.shangar...@gmail.com > > > > >: > > > > > > > > > Vladimir, > > > > > > > > > > I don't see any difference here. > > > > > > > > > > The same possibilities would be available as with normal cache > start: > > > > > -XML; > > > > > -remote node. > > > > > > > > > > >3) Avoid race condition when configuration changes between > > > configuration > > > > > read and method call (what could lead to a number of strange > > effects). > > > > > > > > > > Well, we could add *old* configuration parameter for CAS-like > > semantic. > > > > > > > > > > On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov < > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Ed, > > > > > > > > > > > > Caches in .NET could be started programmatically, from XML which > > .NET > > > > API > > > > > > has no access to, or dynamically from remote nodes (eg Java > node). > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev < > > > > > > eduard.shangar...@gmail.com > > > > > > >: > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > How does .Net user start caches right now? > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov < > > > > voze...@gridgain.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Eduard, > > > > > > > > > > > > > > > > Simple != correct. Let’s consider a simple use case: user > want > > to > > > > > > change > > > > > > > > PARTITIONED -> REPLICATED from .NET, but do not some classes > > from > > > > > > > > CacheConfiguration. How do we solve this? > > > > > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > > > > > > > eduard.shangar...@gmail.com > > > > > > > > >: > > > > > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > > > > > I propose not to change cache configuration in runtime but > > > > restart > > > > > > > cache > > > > > > > > > with the new compatible configuration on data which we have > > > > > > underfoot. > > > > > > > > > > > > > > > > > > What we could change: > > > > > > > > > -backup count; > > > > > > > > > -TRANSACTIONAL <-> ATOMIC; > > > > > > > > > -REPLICATED - PARTITIONED; > > > > > > > > > -other settings. > > > > > > > > > > > > > > > > > > So, yeah, it would be great to have a possibility to change > > > some > > > > > > > > properties > > > > > > > > > in runtime. But right we don't any way to change anything > > > except > > > > > > > indexes > > > > > > > > > and SQL fields. > > > > > > > > > > > > > > > > > > We already have all mechanism to do this. > > > > > > > > > The main issue is to make it reliable and exclude cases > when > > we > > > > > could > > > > > > > > come > > > > > > > > > to the unrecoverable state. > > > > > > > > > > > > > > > > > > So, I suggest keeping the solution as simple as possible. > > > > > > > > > For indexes clashes and ClassNotFoundException we could > > revert > > > > > > > > > configuration update and start with the old configuration. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov < > > > > > > voze...@gridgain.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Eduard, > > > > > > > > > > > > > > > > > > > > Got it. Please take the following things in count during > > > > design: > > > > > > > > > > > > > > > > > > > > 1) Two distinct PMEs might not work well. Consider a > > > situation > > > > > > w1hen > > > > > > > I > >
Re: New API for changing configuration of persistent caches
Vladimir, I didn't get how does .Net user start caches right now? XML and remote node? Right? On Wed, Nov 21, 2018 at 6:55 PM Vladimir Ozerov wrote: > Ed, > > We are not Java product. We support 6 platforms at the moment. Why do we > implement a feature which can only be used in Java, when it is very easy to > make it available from everywhere? > > ср, 21 нояб. 2018 г. в 18:50, Eduard Shangareev >: > > > Vladimir, > > > > It would be Java API specific. > > For a user, we would add a new command for console.sh which would take an > > XML-file path as a parameter. > > > > We could add other possibilities: for example, with the builder which > would > > finally call this Ignite.restartCaches method. But it's nice to have, > not a > > mandatory one. > > > > On Wed, Nov 21, 2018 at 6:43 PM Vladimir Ozerov > > wrote: > > > > > Ed, > > > > > > Could you please demonstrate how .NET node or .NET will change cache > > > configuration with proposed API? Taking in count that XML is not > > available > > > in most cases, and custom Java classes from cache configuration are > > > available only on server nodes and only from Java. > > > > > > ср, 21 нояб. 2018 г. в 18:36, Eduard Shangareev < > > > eduard.shangar...@gmail.com > > > >: > > > > > > > Vladimir, > > > > > > > > I don't see any difference here. > > > > > > > > The same possibilities would be available as with normal cache start: > > > > -XML; > > > > -remote node. > > > > > > > > >3) Avoid race condition when configuration changes between > > configuration > > > > read and method call (what could lead to a number of strange > effects). > > > > > > > > Well, we could add *old* configuration parameter for CAS-like > semantic. > > > > > > > > On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov < > voze...@gridgain.com> > > > > wrote: > > > > > > > > > Ed, > > > > > > > > > > Caches in .NET could be started programmatically, from XML which > .NET > > > API > > > > > has no access to, or dynamically from remote nodes (eg Java node). > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev < > > > > > eduard.shangar...@gmail.com > > > > > >: > > > > > > > > > > > Vladimir, > > > > > > > > > > > > How does .Net user start caches right now? > > > > > > > > > > > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov < > > > voze...@gridgain.com> > > > > > > wrote: > > > > > > > > > > > > > Eduard, > > > > > > > > > > > > > > Simple != correct. Let’s consider a simple use case: user want > to > > > > > change > > > > > > > PARTITIONED -> REPLICATED from .NET, but do not some classes > from > > > > > > > CacheConfiguration. How do we solve this? > > > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > > > > > > eduard.shangar...@gmail.com > > > > > > > >: > > > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > > > I propose not to change cache configuration in runtime but > > > restart > > > > > > cache > > > > > > > > with the new compatible configuration on data which we have > > > > > underfoot. > > > > > > > > > > > > > > > > What we could change: > > > > > > > > -backup count; > > > > > > > > -TRANSACTIONAL <-> ATOMIC; > > > > > > > > -REPLICATED - PARTITIONED; > > > > > > > > -other settings. > > > > > > > > > > > > > > > > So, yeah, it would be great to have a possibility to change > > some > > > > > > > properties > > > > > > > > in runtime. But right we don't any way to change anything > > except > > > > > > indexes > > > > > > > > and SQL fields. > > > > > > > > > > > > > > > > We already have all mechanism to do this. > > > > > > > > The main issue is to make it reliable and exclude cases when > we > > > > could > > > > > > > come > > > > > > > > to the unrecoverable state. > > > > > > > > > > > > > > > > So, I suggest keeping the solution as simple as possible. > > > > > > > > For indexes clashes and ClassNotFoundException we could > revert > > > > > > > > configuration update and start with the old configuration. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov < > > > > > voze...@gridgain.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Eduard, > > > > > > > > > > > > > > > > > > Got it. Please take the following things in count during > > > design: > > > > > > > > > > > > > > > > > > 1) Two distinct PMEs might not work well. Consider a > > situation > > > > > w1hen > > > > > > I > > > > > > > > > decided to move a cache with index "MY_INDEX" from schema A > > to > > > > > schema > > > > > > > B. > > > > > > > > > While cache was stopped, another cache with index > "MY_INDEX" > > > was > > > > > > > created > > > > > > > > in > > > > > > > > > schema B. Now first cache cannot start due to index name > > > > conflict. > > > > > > > > > 2) Cancelling index creation is also bad idea because this > is > > > > > > > potentially > > > > > > > > > long operation. Instead, most likely that we should wait to > > > > > > con
Re: New API for changing configuration of persistent caches
Ed, We are not Java product. We support 6 platforms at the moment. Why do we implement a feature which can only be used in Java, when it is very easy to make it available from everywhere? ср, 21 нояб. 2018 г. в 18:50, Eduard Shangareev : > Vladimir, > > It would be Java API specific. > For a user, we would add a new command for console.sh which would take an > XML-file path as a parameter. > > We could add other possibilities: for example, with the builder which would > finally call this Ignite.restartCaches method. But it's nice to have, not a > mandatory one. > > On Wed, Nov 21, 2018 at 6:43 PM Vladimir Ozerov > wrote: > > > Ed, > > > > Could you please demonstrate how .NET node or .NET will change cache > > configuration with proposed API? Taking in count that XML is not > available > > in most cases, and custom Java classes from cache configuration are > > available only on server nodes and only from Java. > > > > ср, 21 нояб. 2018 г. в 18:36, Eduard Shangareev < > > eduard.shangar...@gmail.com > > >: > > > > > Vladimir, > > > > > > I don't see any difference here. > > > > > > The same possibilities would be available as with normal cache start: > > > -XML; > > > -remote node. > > > > > > >3) Avoid race condition when configuration changes between > configuration > > > read and method call (what could lead to a number of strange effects). > > > > > > Well, we could add *old* configuration parameter for CAS-like semantic. > > > > > > On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov > > > wrote: > > > > > > > Ed, > > > > > > > > Caches in .NET could be started programmatically, from XML which .NET > > API > > > > has no access to, or dynamically from remote nodes (eg Java node). > > > > > > > > ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev < > > > > eduard.shangar...@gmail.com > > > > >: > > > > > > > > > Vladimir, > > > > > > > > > > How does .Net user start caches right now? > > > > > > > > > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov < > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Eduard, > > > > > > > > > > > > Simple != correct. Let’s consider a simple use case: user want to > > > > change > > > > > > PARTITIONED -> REPLICATED from .NET, but do not some classes from > > > > > > CacheConfiguration. How do we solve this? > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > > > > > eduard.shangar...@gmail.com > > > > > > >: > > > > > > > > > > > > > Vladimir, > > > > > > > > > > > > > > I propose not to change cache configuration in runtime but > > restart > > > > > cache > > > > > > > with the new compatible configuration on data which we have > > > > underfoot. > > > > > > > > > > > > > > What we could change: > > > > > > > -backup count; > > > > > > > -TRANSACTIONAL <-> ATOMIC; > > > > > > > -REPLICATED - PARTITIONED; > > > > > > > -other settings. > > > > > > > > > > > > > > So, yeah, it would be great to have a possibility to change > some > > > > > > properties > > > > > > > in runtime. But right we don't any way to change anything > except > > > > > indexes > > > > > > > and SQL fields. > > > > > > > > > > > > > > We already have all mechanism to do this. > > > > > > > The main issue is to make it reliable and exclude cases when we > > > could > > > > > > come > > > > > > > to the unrecoverable state. > > > > > > > > > > > > > > So, I suggest keeping the solution as simple as possible. > > > > > > > For indexes clashes and ClassNotFoundException we could revert > > > > > > > configuration update and start with the old configuration. > > > > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov < > > > > voze...@gridgain.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Eduard, > > > > > > > > > > > > > > > > Got it. Please take the following things in count during > > design: > > > > > > > > > > > > > > > > 1) Two distinct PMEs might not work well. Consider a > situation > > > > w1hen > > > > > I > > > > > > > > decided to move a cache with index "MY_INDEX" from schema A > to > > > > schema > > > > > > B. > > > > > > > > While cache was stopped, another cache with index "MY_INDEX" > > was > > > > > > created > > > > > > > in > > > > > > > > schema B. Now first cache cannot start due to index name > > > conflict. > > > > > > > > 2) Cancelling index creation is also bad idea because this is > > > > > > potentially > > > > > > > > long operation. Instead, most likely that we should wait to > > > > > concurrent > > > > > > > > schema operations to finish first. That is, all operations on > > > cache > > > > > > > should > > > > > > > > be ordered wrt each other somehow > > > > > > > > 3) Why do we think that cache restart will be needed at all? > We > > > > have > > > > > a > > > > > > > lot > > > > > > > > of configuration properties which could be changed safely > > either > > > > > > without > > > > > > > > PME or with a single PME. - rebalance properties, cache store > > >
[GitHub] asfgit closed pull request #76: IGNITE-10275 Refactor of visa caching.
asfgit closed pull request #76: IGNITE-10275 Refactor of visa caching.
URL: https://github.com/apache/ignite-teamcity-bot/pull/76
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
index c7f0e11a..ea8aa9bd 100644
---
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
+++
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
@@ -79,6 +79,7 @@
import org.apache.ignite.ci.util.CacheUpdateUtil;
import org.apache.ignite.ci.util.CollectionUtil;
import org.apache.ignite.ci.util.ObjectInterner;
+import org.apache.ignite.ci.web.model.hist.VisasHistoryStorage;
import org.apache.ignite.ci.web.rest.parms.FullQueryParams;
import org.jetbrains.annotations.NotNull;
@@ -116,6 +117,11 @@
@Inject
private Ignite ignite;
+
+/** */
+@Inject
+private VisasHistoryStorage visasHistStorage;
+
/**
* Teamcity
*/
@@ -158,7 +164,8 @@
buildsCache(), this::addBuildOccurrenceToFailuresStat,
buildsFailureRunStatCache(), testRunStatCache(),
testFullCache(),
-buildProblemsCache());
+buildProblemsCache(),
+visasHistStorage.visas());
}
@Override
diff --git
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/db/DbMigrations.java
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/db/DbMigrations.java
index a8307ffe..cc64e34d 100644
---
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/db/DbMigrations.java
+++
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/db/DbMigrations.java
@@ -17,33 +17,38 @@
package org.apache.ignite.ci.db;
+import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.function.Consumer;
import javax.cache.Cache;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.binary.BinaryObject;
import org.apache.ignite.cache.CacheAtomicityMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.ci.ITeamcity;
import org.apache.ignite.ci.IgnitePersistentTeamcity;
-import org.apache.ignite.ci.analysis.Expirable;
import org.apache.ignite.ci.analysis.RunStat;
import org.apache.ignite.ci.analysis.SuiteInBranch;
import org.apache.ignite.ci.analysis.TestInBranch;
import org.apache.ignite.ci.issue.Issue;
import org.apache.ignite.ci.issue.IssueKey;
import org.apache.ignite.ci.issue.IssuesStorage;
-import org.apache.ignite.ci.tcmodel.hist.BuildRef;
+import org.apache.ignite.ci.observer.CompactBuildsInfo;
import org.apache.ignite.ci.tcmodel.result.Build;
import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrences;
import org.apache.ignite.ci.tcmodel.result.stat.Statistics;
import org.apache.ignite.ci.tcmodel.result.tests.TestOccurrenceFull;
import org.apache.ignite.ci.tcmodel.result.tests.TestOccurrences;
+import org.apache.ignite.ci.web.model.CompactContributionKey;
+import org.apache.ignite.ci.web.model.CompactVisa;
+import org.apache.ignite.ci.web.model.CompactVisaRequest;
+import org.apache.ignite.ci.web.model.hist.VisasHistoryStorage;
import org.apache.ignite.ci.web.rest.build.GetBuildTestFailures;
-import org.apache.ignite.ci.web.rest.pr.GetPrTestFailures;
import org.apache.ignite.ci.web.rest.tracked.GetTrackedBranchTestResults;
import org.apache.ignite.configuration.CacheConfiguration;
import org.slf4j.Logger;
@@ -104,6 +109,9 @@
@Deprecated
public static final String TEAMCITY_BUILD_CACHE_NAME_OLD = "teamcityBuild";
+/** */
+@Deprecated
+public static final String COMPACT_VISAS_HISTORY_CACHE_NAME =
"compactVisasHistoryCache";
private static final String CHANGE_INFO_FULL = "changeInfoFull";
private static final String CHANGES_LIST = "changesList";
@@ -131,10 +139,45 @@ public void dataMigration(
IgniteCache suiteHistCache,
IgniteCache testHistCache,
Cache testFullCache,
-Cache problemsCache) {
+Cache problemsCache,
+Cache>
visasCache) {
doneMigrations = doneMigrationsCache();
+applyMigration(COMPACT_VISAS_HISTORY_CACHE_NAME + "-to-" +
VisasHistoryStorage.VISAS_CACHE_NAME, () -> {
+IgniteCache oldVisasCache =
ignite.cache(COMPACT_VISAS_HISTORY_CACHE_NAME).withKeepBinary();
+
+if (Objects.isNull(oldVisasCache)) {
+System.out.println("Old cache [" +
COMPACT_VISAS_HISTORY_C
Re: New API for changing configuration of persistent caches
Vladimir, It would be Java API specific. For a user, we would add a new command for console.sh which would take an XML-file path as a parameter. We could add other possibilities: for example, with the builder which would finally call this Ignite.restartCaches method. But it's nice to have, not a mandatory one. On Wed, Nov 21, 2018 at 6:43 PM Vladimir Ozerov wrote: > Ed, > > Could you please demonstrate how .NET node or .NET will change cache > configuration with proposed API? Taking in count that XML is not available > in most cases, and custom Java classes from cache configuration are > available only on server nodes and only from Java. > > ср, 21 нояб. 2018 г. в 18:36, Eduard Shangareev < > eduard.shangar...@gmail.com > >: > > > Vladimir, > > > > I don't see any difference here. > > > > The same possibilities would be available as with normal cache start: > > -XML; > > -remote node. > > > > >3) Avoid race condition when configuration changes between configuration > > read and method call (what could lead to a number of strange effects). > > > > Well, we could add *old* configuration parameter for CAS-like semantic. > > > > On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov > > wrote: > > > > > Ed, > > > > > > Caches in .NET could be started programmatically, from XML which .NET > API > > > has no access to, or dynamically from remote nodes (eg Java node). > > > > > > ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev < > > > eduard.shangar...@gmail.com > > > >: > > > > > > > Vladimir, > > > > > > > > How does .Net user start caches right now? > > > > > > > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov < > voze...@gridgain.com> > > > > wrote: > > > > > > > > > Eduard, > > > > > > > > > > Simple != correct. Let’s consider a simple use case: user want to > > > change > > > > > PARTITIONED -> REPLICATED from .NET, but do not some classes from > > > > > CacheConfiguration. How do we solve this? > > > > > > > > > > Vladimir. > > > > > > > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > > > > eduard.shangar...@gmail.com > > > > > >: > > > > > > > > > > > Vladimir, > > > > > > > > > > > > I propose not to change cache configuration in runtime but > restart > > > > cache > > > > > > with the new compatible configuration on data which we have > > > underfoot. > > > > > > > > > > > > What we could change: > > > > > > -backup count; > > > > > > -TRANSACTIONAL <-> ATOMIC; > > > > > > -REPLICATED - PARTITIONED; > > > > > > -other settings. > > > > > > > > > > > > So, yeah, it would be great to have a possibility to change some > > > > > properties > > > > > > in runtime. But right we don't any way to change anything except > > > > indexes > > > > > > and SQL fields. > > > > > > > > > > > > We already have all mechanism to do this. > > > > > > The main issue is to make it reliable and exclude cases when we > > could > > > > > come > > > > > > to the unrecoverable state. > > > > > > > > > > > > So, I suggest keeping the solution as simple as possible. > > > > > > For indexes clashes and ClassNotFoundException we could revert > > > > > > configuration update and start with the old configuration. > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov < > > > voze...@gridgain.com> > > > > > > wrote: > > > > > > > > > > > > > Eduard, > > > > > > > > > > > > > > Got it. Please take the following things in count during > design: > > > > > > > > > > > > > > 1) Two distinct PMEs might not work well. Consider a situation > > > w1hen > > > > I > > > > > > > decided to move a cache with index "MY_INDEX" from schema A to > > > schema > > > > > B. > > > > > > > While cache was stopped, another cache with index "MY_INDEX" > was > > > > > created > > > > > > in > > > > > > > schema B. Now first cache cannot start due to index name > > conflict. > > > > > > > 2) Cancelling index creation is also bad idea because this is > > > > > potentially > > > > > > > long operation. Instead, most likely that we should wait to > > > > concurrent > > > > > > > schema operations to finish first. That is, all operations on > > cache > > > > > > should > > > > > > > be ordered wrt each other somehow > > > > > > > 3) Why do we think that cache restart will be needed at all? We > > > have > > > > a > > > > > > lot > > > > > > > of configuration properties which could be changed safely > either > > > > > without > > > > > > > PME or with a single PME. - rebalance properties, cache store > > > > > properties > > > > > > > (especially write-behind stuff), some query properties (e.g. > > > "detail > > > > > > > metrics"), etc.. In essence, it seems that >50% of properties > > could > > > > be > > > > > > > changed without cache restart, other 25% will not be supported, > > and > > > > the > > > > > > > rest may require restart. > > > > > > > 4) Client nodes and thin client may not have necessary classes > in > > > > > > > classpath. E.g. consider a user which want to change rebalance > > > > timeout > > > > > > fo
Re: New API for changing configuration of persistent caches
Ed, Could you please demonstrate how .NET node or .NET will change cache configuration with proposed API? Taking in count that XML is not available in most cases, and custom Java classes from cache configuration are available only on server nodes and only from Java. ср, 21 нояб. 2018 г. в 18:36, Eduard Shangareev : > Vladimir, > > I don't see any difference here. > > The same possibilities would be available as with normal cache start: > -XML; > -remote node. > > >3) Avoid race condition when configuration changes between configuration > read and method call (what could lead to a number of strange effects). > > Well, we could add *old* configuration parameter for CAS-like semantic. > > On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov > wrote: > > > Ed, > > > > Caches in .NET could be started programmatically, from XML which .NET API > > has no access to, or dynamically from remote nodes (eg Java node). > > > > ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev < > > eduard.shangar...@gmail.com > > >: > > > > > Vladimir, > > > > > > How does .Net user start caches right now? > > > > > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov > > > wrote: > > > > > > > Eduard, > > > > > > > > Simple != correct. Let’s consider a simple use case: user want to > > change > > > > PARTITIONED -> REPLICATED from .NET, but do not some classes from > > > > CacheConfiguration. How do we solve this? > > > > > > > > Vladimir. > > > > > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > > > eduard.shangar...@gmail.com > > > > >: > > > > > > > > > Vladimir, > > > > > > > > > > I propose not to change cache configuration in runtime but restart > > > cache > > > > > with the new compatible configuration on data which we have > > underfoot. > > > > > > > > > > What we could change: > > > > > -backup count; > > > > > -TRANSACTIONAL <-> ATOMIC; > > > > > -REPLICATED - PARTITIONED; > > > > > -other settings. > > > > > > > > > > So, yeah, it would be great to have a possibility to change some > > > > properties > > > > > in runtime. But right we don't any way to change anything except > > > indexes > > > > > and SQL fields. > > > > > > > > > > We already have all mechanism to do this. > > > > > The main issue is to make it reliable and exclude cases when we > could > > > > come > > > > > to the unrecoverable state. > > > > > > > > > > So, I suggest keeping the solution as simple as possible. > > > > > For indexes clashes and ClassNotFoundException we could revert > > > > > configuration update and start with the old configuration. > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov < > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Eduard, > > > > > > > > > > > > Got it. Please take the following things in count during design: > > > > > > > > > > > > 1) Two distinct PMEs might not work well. Consider a situation > > w1hen > > > I > > > > > > decided to move a cache with index "MY_INDEX" from schema A to > > schema > > > > B. > > > > > > While cache was stopped, another cache with index "MY_INDEX" was > > > > created > > > > > in > > > > > > schema B. Now first cache cannot start due to index name > conflict. > > > > > > 2) Cancelling index creation is also bad idea because this is > > > > potentially > > > > > > long operation. Instead, most likely that we should wait to > > > concurrent > > > > > > schema operations to finish first. That is, all operations on > cache > > > > > should > > > > > > be ordered wrt each other somehow > > > > > > 3) Why do we think that cache restart will be needed at all? We > > have > > > a > > > > > lot > > > > > > of configuration properties which could be changed safely either > > > > without > > > > > > PME or with a single PME. - rebalance properties, cache store > > > > properties > > > > > > (especially write-behind stuff), some query properties (e.g. > > "detail > > > > > > metrics"), etc.. In essence, it seems that >50% of properties > could > > > be > > > > > > changed without cache restart, other 25% will not be supported, > and > > > the > > > > > > rest may require restart. > > > > > > 4) Client nodes and thin client may not have necessary classes in > > > > > > classpath. E.g. consider a user which want to change rebalance > > > timeout > > > > > for > > > > > > cache, but do not have configured interceptor in classpath. With > > > > proposed > > > > > > API it will be impossible. This is especially true for non-Java > > > > clients. > > > > > > > > > > > > That said, I think we should consider another API which will not > > > > require > > > > > > full CacheConfiguration object. This might be a kind of builder > or > > > so. > > > > > And > > > > > > once user set properties he want to change to the builder, we can > > > > analyze > > > > > > them and either change them in runtime without PME, change with a > > > > single > > > > > > PME or change with full cache restart. > > > > > > > > > > > > What do you think? > > > > > > > > > > > > V
Re: New API for changing configuration of persistent caches
Vladimir, I don't see any difference here. The same possibilities would be available as with normal cache start: -XML; -remote node. >3) Avoid race condition when configuration changes between configuration read and method call (what could lead to a number of strange effects). Well, we could add *old* configuration parameter for CAS-like semantic. On Wed, Nov 21, 2018 at 6:26 PM Vladimir Ozerov wrote: > Ed, > > Caches in .NET could be started programmatically, from XML which .NET API > has no access to, or dynamically from remote nodes (eg Java node). > > ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev < > eduard.shangar...@gmail.com > >: > > > Vladimir, > > > > How does .Net user start caches right now? > > > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov > > wrote: > > > > > Eduard, > > > > > > Simple != correct. Let’s consider a simple use case: user want to > change > > > PARTITIONED -> REPLICATED from .NET, but do not some classes from > > > CacheConfiguration. How do we solve this? > > > > > > Vladimir. > > > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > > eduard.shangar...@gmail.com > > > >: > > > > > > > Vladimir, > > > > > > > > I propose not to change cache configuration in runtime but restart > > cache > > > > with the new compatible configuration on data which we have > underfoot. > > > > > > > > What we could change: > > > > -backup count; > > > > -TRANSACTIONAL <-> ATOMIC; > > > > -REPLICATED - PARTITIONED; > > > > -other settings. > > > > > > > > So, yeah, it would be great to have a possibility to change some > > > properties > > > > in runtime. But right we don't any way to change anything except > > indexes > > > > and SQL fields. > > > > > > > > We already have all mechanism to do this. > > > > The main issue is to make it reliable and exclude cases when we could > > > come > > > > to the unrecoverable state. > > > > > > > > So, I suggest keeping the solution as simple as possible. > > > > For indexes clashes and ClassNotFoundException we could revert > > > > configuration update and start with the old configuration. > > > > > > > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov < > voze...@gridgain.com> > > > > wrote: > > > > > > > > > Eduard, > > > > > > > > > > Got it. Please take the following things in count during design: > > > > > > > > > > 1) Two distinct PMEs might not work well. Consider a situation > w1hen > > I > > > > > decided to move a cache with index "MY_INDEX" from schema A to > schema > > > B. > > > > > While cache was stopped, another cache with index "MY_INDEX" was > > > created > > > > in > > > > > schema B. Now first cache cannot start due to index name conflict. > > > > > 2) Cancelling index creation is also bad idea because this is > > > potentially > > > > > long operation. Instead, most likely that we should wait to > > concurrent > > > > > schema operations to finish first. That is, all operations on cache > > > > should > > > > > be ordered wrt each other somehow > > > > > 3) Why do we think that cache restart will be needed at all? We > have > > a > > > > lot > > > > > of configuration properties which could be changed safely either > > > without > > > > > PME or with a single PME. - rebalance properties, cache store > > > properties > > > > > (especially write-behind stuff), some query properties (e.g. > "detail > > > > > metrics"), etc.. In essence, it seems that >50% of properties could > > be > > > > > changed without cache restart, other 25% will not be supported, and > > the > > > > > rest may require restart. > > > > > 4) Client nodes and thin client may not have necessary classes in > > > > > classpath. E.g. consider a user which want to change rebalance > > timeout > > > > for > > > > > cache, but do not have configured interceptor in classpath. With > > > proposed > > > > > API it will be impossible. This is especially true for non-Java > > > clients. > > > > > > > > > > That said, I think we should consider another API which will not > > > require > > > > > full CacheConfiguration object. This might be a kind of builder or > > so. > > > > And > > > > > once user set properties he want to change to the builder, we can > > > analyze > > > > > them and either change them in runtime without PME, change with a > > > single > > > > > PME or change with full cache restart. > > > > > > > > > > What do you think? > > > > > > > > > > Vladimir. > > > > > > > > > > On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev < > > > > > eshangar...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Vladimir, > > > > > > > > > > > > 1) Affinity could be changed, but count of partition couldn't be. > > > > > > 2) So it would trigger 2 PME. Dynamic start and stop. > > > > > > 3) In theory, should cancel them and new setting should be > applied. > > > How > > > > > it > > > > > > works now? Create an index and stop node, for example. > > > > > > > > > > > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov < > > > voze...@gridgain.com> > >
Re: New API for changing configuration of persistent caches
Ed, Caches in .NET could be started programmatically, from XML which .NET API has no access to, or dynamically from remote nodes (eg Java node). ср, 21 нояб. 2018 г. в 18:24, Eduard Shangareev : > Vladimir, > > How does .Net user start caches right now? > > On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov > wrote: > > > Eduard, > > > > Simple != correct. Let’s consider a simple use case: user want to change > > PARTITIONED -> REPLICATED from .NET, but do not some classes from > > CacheConfiguration. How do we solve this? > > > > Vladimir. > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > eduard.shangar...@gmail.com > > >: > > > > > Vladimir, > > > > > > I propose not to change cache configuration in runtime but restart > cache > > > with the new compatible configuration on data which we have underfoot. > > > > > > What we could change: > > > -backup count; > > > -TRANSACTIONAL <-> ATOMIC; > > > -REPLICATED - PARTITIONED; > > > -other settings. > > > > > > So, yeah, it would be great to have a possibility to change some > > properties > > > in runtime. But right we don't any way to change anything except > indexes > > > and SQL fields. > > > > > > We already have all mechanism to do this. > > > The main issue is to make it reliable and exclude cases when we could > > come > > > to the unrecoverable state. > > > > > > So, I suggest keeping the solution as simple as possible. > > > For indexes clashes and ClassNotFoundException we could revert > > > configuration update and start with the old configuration. > > > > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov > > > wrote: > > > > > > > Eduard, > > > > > > > > Got it. Please take the following things in count during design: > > > > > > > > 1) Two distinct PMEs might not work well. Consider a situation w1hen > I > > > > decided to move a cache with index "MY_INDEX" from schema A to schema > > B. > > > > While cache was stopped, another cache with index "MY_INDEX" was > > created > > > in > > > > schema B. Now first cache cannot start due to index name conflict. > > > > 2) Cancelling index creation is also bad idea because this is > > potentially > > > > long operation. Instead, most likely that we should wait to > concurrent > > > > schema operations to finish first. That is, all operations on cache > > > should > > > > be ordered wrt each other somehow > > > > 3) Why do we think that cache restart will be needed at all? We have > a > > > lot > > > > of configuration properties which could be changed safely either > > without > > > > PME or with a single PME. - rebalance properties, cache store > > properties > > > > (especially write-behind stuff), some query properties (e.g. "detail > > > > metrics"), etc.. In essence, it seems that >50% of properties could > be > > > > changed without cache restart, other 25% will not be supported, and > the > > > > rest may require restart. > > > > 4) Client nodes and thin client may not have necessary classes in > > > > classpath. E.g. consider a user which want to change rebalance > timeout > > > for > > > > cache, but do not have configured interceptor in classpath. With > > proposed > > > > API it will be impossible. This is especially true for non-Java > > clients. > > > > > > > > That said, I think we should consider another API which will not > > require > > > > full CacheConfiguration object. This might be a kind of builder or > so. > > > And > > > > once user set properties he want to change to the builder, we can > > analyze > > > > them and either change them in runtime without PME, change with a > > single > > > > PME or change with full cache restart. > > > > > > > > What do you think? > > > > > > > > Vladimir. > > > > > > > > On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev < > > > > eshangar...@gridgain.com> > > > > wrote: > > > > > > > > > Vladimir, > > > > > > > > > > 1) Affinity could be changed, but count of partition couldn't be. > > > > > 2) So it would trigger 2 PME. Dynamic start and stop. > > > > > 3) In theory, should cancel them and new setting should be applied. > > How > > > > it > > > > > works now? Create an index and stop node, for example. > > > > > > > > > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov < > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Hi Ed, > > > > > > > > > > > > Several questions from my side: > > > > > > 1) If we do not allow to change the most demanded by users things > > > like > > > > > > affinity or persistence/in-memory, then what kind of > configuration > > > > > > properties do we expect to be changed? Can we have several > > examples? > > > > > > 2) How will it interact with PME? > > > > > > 3) How will it interact with CREATE INDEX and ALTER TABLE > commands? > > > > > > > > > > > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < > > > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > > > > > Igniters, > > > > > > > > > > > > > > I propose new public API to change the cache configuration of > > > >
[GitHub] ignite pull request #5396: IGNITE-10226 Fixed wrong partition state recovery
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5396 ---
Re: New API for changing configuration of persistent caches
Vladimir, How does .Net user start caches right now? On Wed, Nov 21, 2018 at 6:10 PM Vladimir Ozerov wrote: > Eduard, > > Simple != correct. Let’s consider a simple use case: user want to change > PARTITIONED -> REPLICATED from .NET, but do not some classes from > CacheConfiguration. How do we solve this? > > Vladimir. > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > eduard.shangar...@gmail.com > >: > > > Vladimir, > > > > I propose not to change cache configuration in runtime but restart cache > > with the new compatible configuration on data which we have underfoot. > > > > What we could change: > > -backup count; > > -TRANSACTIONAL <-> ATOMIC; > > -REPLICATED - PARTITIONED; > > -other settings. > > > > So, yeah, it would be great to have a possibility to change some > properties > > in runtime. But right we don't any way to change anything except indexes > > and SQL fields. > > > > We already have all mechanism to do this. > > The main issue is to make it reliable and exclude cases when we could > come > > to the unrecoverable state. > > > > So, I suggest keeping the solution as simple as possible. > > For indexes clashes and ClassNotFoundException we could revert > > configuration update and start with the old configuration. > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov > > wrote: > > > > > Eduard, > > > > > > Got it. Please take the following things in count during design: > > > > > > 1) Two distinct PMEs might not work well. Consider a situation w1hen I > > > decided to move a cache with index "MY_INDEX" from schema A to schema > B. > > > While cache was stopped, another cache with index "MY_INDEX" was > created > > in > > > schema B. Now first cache cannot start due to index name conflict. > > > 2) Cancelling index creation is also bad idea because this is > potentially > > > long operation. Instead, most likely that we should wait to concurrent > > > schema operations to finish first. That is, all operations on cache > > should > > > be ordered wrt each other somehow > > > 3) Why do we think that cache restart will be needed at all? We have a > > lot > > > of configuration properties which could be changed safely either > without > > > PME or with a single PME. - rebalance properties, cache store > properties > > > (especially write-behind stuff), some query properties (e.g. "detail > > > metrics"), etc.. In essence, it seems that >50% of properties could be > > > changed without cache restart, other 25% will not be supported, and the > > > rest may require restart. > > > 4) Client nodes and thin client may not have necessary classes in > > > classpath. E.g. consider a user which want to change rebalance timeout > > for > > > cache, but do not have configured interceptor in classpath. With > proposed > > > API it will be impossible. This is especially true for non-Java > clients. > > > > > > That said, I think we should consider another API which will not > require > > > full CacheConfiguration object. This might be a kind of builder or so. > > And > > > once user set properties he want to change to the builder, we can > analyze > > > them and either change them in runtime without PME, change with a > single > > > PME or change with full cache restart. > > > > > > What do you think? > > > > > > Vladimir. > > > > > > On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev < > > > eshangar...@gridgain.com> > > > wrote: > > > > > > > Vladimir, > > > > > > > > 1) Affinity could be changed, but count of partition couldn't be. > > > > 2) So it would trigger 2 PME. Dynamic start and stop. > > > > 3) In theory, should cancel them and new setting should be applied. > How > > > it > > > > works now? Create an index and stop node, for example. > > > > > > > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov < > voze...@gridgain.com> > > > > wrote: > > > > > > > > > Hi Ed, > > > > > > > > > > Several questions from my side: > > > > > 1) If we do not allow to change the most demanded by users things > > like > > > > > affinity or persistence/in-memory, then what kind of configuration > > > > > properties do we expect to be changed? Can we have several > examples? > > > > > 2) How will it interact with PME? > > > > > 3) How will it interact with CREATE INDEX and ALTER TABLE commands? > > > > > > > > > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < > > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > > > Igniters, > > > > > > > > > > > > I propose new public API to change the cache configuration of > > > > persistent > > > > > > caches with keeping data. > > > > > > > > > > > > It would look like: > > > > > > > > > > > > Ignite ignite = ...; > > > > > > ignite.restartCaches(cfg1, ... cfgN); > > > > > > > > > > > > where cfgX is a new cache configuration, which contains the name > of > > > an > > > > > > existing persistent cache. > > > > > > > > > > > > The obvious limitation: > > > > > > - affinity key mapping couldn't be changed; > > > > > > - count of partitions couldn't be
Re: New API for changing configuration of persistent caches
Correction: “do not *have* some classes”. ср, 21 нояб. 2018 г. в 18:09, Vladimir Ozerov : > Eduard, > > Simple != correct. Let’s consider a simple use case: user want to change > PARTITIONED -> REPLICATED from .NET, but do not some classes from > CacheConfiguration. How do we solve this? > > Vladimir. > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > eduard.shangar...@gmail.com>: > >> Vladimir, >> >> I propose not to change cache configuration in runtime but restart cache >> with the new compatible configuration on data which we have underfoot. >> >> What we could change: >> -backup count; >> -TRANSACTIONAL <-> ATOMIC; >> -REPLICATED - PARTITIONED; >> -other settings. >> >> So, yeah, it would be great to have a possibility to change some >> properties >> in runtime. But right we don't any way to change anything except indexes >> and SQL fields. >> >> We already have all mechanism to do this. >> The main issue is to make it reliable and exclude cases when we could come >> to the unrecoverable state. >> >> So, I suggest keeping the solution as simple as possible. >> For indexes clashes and ClassNotFoundException we could revert >> configuration update and start with the old configuration. >> >> >> On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov >> wrote: >> >> > Eduard, >> > >> > Got it. Please take the following things in count during design: >> > >> > 1) Two distinct PMEs might not work well. Consider a situation w1hen I >> > decided to move a cache with index "MY_INDEX" from schema A to schema B. >> > While cache was stopped, another cache with index "MY_INDEX" was >> created in >> > schema B. Now first cache cannot start due to index name conflict. >> > 2) Cancelling index creation is also bad idea because this is >> potentially >> > long operation. Instead, most likely that we should wait to concurrent >> > schema operations to finish first. That is, all operations on cache >> should >> > be ordered wrt each other somehow >> > 3) Why do we think that cache restart will be needed at all? We have a >> lot >> > of configuration properties which could be changed safely either without >> > PME or with a single PME. - rebalance properties, cache store properties >> > (especially write-behind stuff), some query properties (e.g. "detail >> > metrics"), etc.. In essence, it seems that >50% of properties could be >> > changed without cache restart, other 25% will not be supported, and the >> > rest may require restart. >> > 4) Client nodes and thin client may not have necessary classes in >> > classpath. E.g. consider a user which want to change rebalance timeout >> for >> > cache, but do not have configured interceptor in classpath. With >> proposed >> > API it will be impossible. This is especially true for non-Java clients. >> > >> > That said, I think we should consider another API which will not require >> > full CacheConfiguration object. This might be a kind of builder or so. >> And >> > once user set properties he want to change to the builder, we can >> analyze >> > them and either change them in runtime without PME, change with a single >> > PME or change with full cache restart. >> > >> > What do you think? >> > >> > Vladimir. >> > >> > On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev < >> > eshangar...@gridgain.com> >> > wrote: >> > >> > > Vladimir, >> > > >> > > 1) Affinity could be changed, but count of partition couldn't be. >> > > 2) So it would trigger 2 PME. Dynamic start and stop. >> > > 3) In theory, should cancel them and new setting should be applied. >> How >> > it >> > > works now? Create an index and stop node, for example. >> > > >> > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov > > >> > > wrote: >> > > >> > > > Hi Ed, >> > > > >> > > > Several questions from my side: >> > > > 1) If we do not allow to change the most demanded by users things >> like >> > > > affinity or persistence/in-memory, then what kind of configuration >> > > > properties do we expect to be changed? Can we have several examples? >> > > > 2) How will it interact with PME? >> > > > 3) How will it interact with CREATE INDEX and ALTER TABLE commands? >> > > > >> > > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < >> > > > eduard.shangar...@gmail.com> wrote: >> > > > >> > > > > Igniters, >> > > > > >> > > > > I propose new public API to change the cache configuration of >> > > persistent >> > > > > caches with keeping data. >> > > > > >> > > > > It would look like: >> > > > > >> > > > > Ignite ignite = ...; >> > > > > ignite.restartCaches(cfg1, ... cfgN); >> > > > > >> > > > > where cfgX is a new cache configuration, which contains the name >> of >> > an >> > > > > existing persistent cache. >> > > > > >> > > > > The obvious limitation: >> > > > > - affinity key mapping couldn't be changed; >> > > > > - count of partitions couldn't be changed; >> > > > > - MVCC couldn't be turned off/on; >> > > > > - persistent couldn't be turned off; >> > > > > - group settings couldn't be changed (group name); >> >
[GitHub] ignite pull request #5464: IGNITE-9902: ScanQuery doesn't take lost partitio...
GitHub user avplatonov opened a pull request: https://github.com/apache/ignite/pull/5464 IGNITE-9902: ScanQuery doesn't take lost partitions into account You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9902 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5464.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5464 commit 832762230a79506096c7ba9c70c10059aeee163e Author: Alexey Platonov Date: 2018-11-21T15:24:03Z init commit ---
[GitHub] ignite pull request #5463: IGNITE-10301 GridToStringBuilder is broken for cl...
GitHub user SomeFire opened a pull request: https://github.com/apache/ignite/pull/5463 IGNITE-10301 GridToStringBuilder is broken for classes with inheritance You can merge this pull request into a Git repository by running: $ git pull https://github.com/SomeFire/ignite IGNITE-10301 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5463.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5463 commit 3432b4bd266b87a728e6c734c337272cf137e3c6 Author: Dmitrii Ryabov Date: 2018-11-21T14:43:49Z IGNITE-10301 GridToStringBuilder is broken for classes with inheritance ---
Re: New API for changing configuration of persistent caches
Alex, Spring XML is not easily available from API. Sometimes it simply doesn’t exist. Why not use builder instead? Ignite.changeCache(“my_cache”, Builder.create().setCacheMode(REPLICATED).build()) Advantages: 1) Available on all platforms 2) Expose only properties we support 3) Avoid race condition when configuration changes between configuration read and method call (what could lead to a number of strange effects). Makes sense? ср, 21 нояб. 2018 г. в 18:16, Alexey Kuznetsov : > Vova, Ed > > I think in case when classes not available we can sent a string with cache > XML configuration (bean) to server node that has all classes in classpath. > > What do you think? > > On Wed, Nov 21, 2018 at 10:10 PM Vladimir Ozerov > wrote: > > > Eduard, > > > > Simple != correct. Let’s consider a simple use case: user want to change > > PARTITIONED -> REPLICATED from .NET, but do not some classes from > > CacheConfiguration. How do we solve this? > > > > Vladimir. > > > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > > eduard.shangar...@gmail.com > > >: > > > > > Vladimir, > > > > > > I propose not to change cache configuration in runtime but restart > cache > > > with the new compatible configuration on data which we have underfoot. > > > > > > What we could change: > > > -backup count; > > > -TRANSACTIONAL <-> ATOMIC; > > > -REPLICATED - PARTITIONED; > > > -other settings. > > > > > > So, yeah, it would be great to have a possibility to change some > > properties > > > in runtime. But right we don't any way to change anything except > indexes > > > and SQL fields. > > > > > > We already have all mechanism to do this. > > > The main issue is to make it reliable and exclude cases when we could > > come > > > to the unrecoverable state. > > > > > > So, I suggest keeping the solution as simple as possible. > > > For indexes clashes and ClassNotFoundException we could revert > > > configuration update and start with the old configuration. > > > > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov > > > wrote: > > > > > > > Eduard, > > > > > > > > Got it. Please take the following things in count during design: > > > > > > > > 1) Two distinct PMEs might not work well. Consider a situation w1hen > I > > > > decided to move a cache with index "MY_INDEX" from schema A to schema > > B. > > > > While cache was stopped, another cache with index "MY_INDEX" was > > created > > > in > > > > schema B. Now first cache cannot start due to index name conflict. > > > > 2) Cancelling index creation is also bad idea because this is > > potentially > > > > long operation. Instead, most likely that we should wait to > concurrent > > > > schema operations to finish first. That is, all operations on cache > > > should > > > > be ordered wrt each other somehow > > > > 3) Why do we think that cache restart will be needed at all? We have > a > > > lot > > > > of configuration properties which could be changed safely either > > without > > > > PME or with a single PME. - rebalance properties, cache store > > properties > > > > (especially write-behind stuff), some query properties (e.g. "detail > > > > metrics"), etc.. In essence, it seems that >50% of properties could > be > > > > changed without cache restart, other 25% will not be supported, and > the > > > > rest may require restart. > > > > 4) Client nodes and thin client may not have necessary classes in > > > > classpath. E.g. consider a user which want to change rebalance > timeout > > > for > > > > cache, but do not have configured interceptor in classpath. With > > proposed > > > > API it will be impossible. This is especially true for non-Java > > clients. > > > > > > > > That said, I think we should consider another API which will not > > require > > > > full CacheConfiguration object. This might be a kind of builder or > so. > > > And > > > > once user set properties he want to change to the builder, we can > > analyze > > > > them and either change them in runtime without PME, change with a > > single > > > > PME or change with full cache restart. > > > > > > > > What do you think? > > > > > > > > Vladimir. > > > > > > > > On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev < > > > > eshangar...@gridgain.com> > > > > wrote: > > > > > > > > > Vladimir, > > > > > > > > > > 1) Affinity could be changed, but count of partition couldn't be. > > > > > 2) So it would trigger 2 PME. Dynamic start and stop. > > > > > 3) In theory, should cancel them and new setting should be applied. > > How > > > > it > > > > > works now? Create an index and stop node, for example. > > > > > > > > > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov < > > voze...@gridgain.com> > > > > > wrote: > > > > > > > > > > > Hi Ed, > > > > > > > > > > > > Several questions from my side: > > > > > > 1) If we do not allow to change the most demanded by users things > > > like > > > > > > affinity or persistence/in-memory, then what kind of > configuration > > > > > > properties do we expect to be changed
Re: New API for changing configuration of persistent caches
Vova, Ed I think in case when classes not available we can sent a string with cache XML configuration (bean) to server node that has all classes in classpath. What do you think? On Wed, Nov 21, 2018 at 10:10 PM Vladimir Ozerov wrote: > Eduard, > > Simple != correct. Let’s consider a simple use case: user want to change > PARTITIONED -> REPLICATED from .NET, but do not some classes from > CacheConfiguration. How do we solve this? > > Vladimir. > > ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev < > eduard.shangar...@gmail.com > >: > > > Vladimir, > > > > I propose not to change cache configuration in runtime but restart cache > > with the new compatible configuration on data which we have underfoot. > > > > What we could change: > > -backup count; > > -TRANSACTIONAL <-> ATOMIC; > > -REPLICATED - PARTITIONED; > > -other settings. > > > > So, yeah, it would be great to have a possibility to change some > properties > > in runtime. But right we don't any way to change anything except indexes > > and SQL fields. > > > > We already have all mechanism to do this. > > The main issue is to make it reliable and exclude cases when we could > come > > to the unrecoverable state. > > > > So, I suggest keeping the solution as simple as possible. > > For indexes clashes and ClassNotFoundException we could revert > > configuration update and start with the old configuration. > > > > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov > > wrote: > > > > > Eduard, > > > > > > Got it. Please take the following things in count during design: > > > > > > 1) Two distinct PMEs might not work well. Consider a situation w1hen I > > > decided to move a cache with index "MY_INDEX" from schema A to schema > B. > > > While cache was stopped, another cache with index "MY_INDEX" was > created > > in > > > schema B. Now first cache cannot start due to index name conflict. > > > 2) Cancelling index creation is also bad idea because this is > potentially > > > long operation. Instead, most likely that we should wait to concurrent > > > schema operations to finish first. That is, all operations on cache > > should > > > be ordered wrt each other somehow > > > 3) Why do we think that cache restart will be needed at all? We have a > > lot > > > of configuration properties which could be changed safely either > without > > > PME or with a single PME. - rebalance properties, cache store > properties > > > (especially write-behind stuff), some query properties (e.g. "detail > > > metrics"), etc.. In essence, it seems that >50% of properties could be > > > changed without cache restart, other 25% will not be supported, and the > > > rest may require restart. > > > 4) Client nodes and thin client may not have necessary classes in > > > classpath. E.g. consider a user which want to change rebalance timeout > > for > > > cache, but do not have configured interceptor in classpath. With > proposed > > > API it will be impossible. This is especially true for non-Java > clients. > > > > > > That said, I think we should consider another API which will not > require > > > full CacheConfiguration object. This might be a kind of builder or so. > > And > > > once user set properties he want to change to the builder, we can > analyze > > > them and either change them in runtime without PME, change with a > single > > > PME or change with full cache restart. > > > > > > What do you think? > > > > > > Vladimir. > > > > > > On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev < > > > eshangar...@gridgain.com> > > > wrote: > > > > > > > Vladimir, > > > > > > > > 1) Affinity could be changed, but count of partition couldn't be. > > > > 2) So it would trigger 2 PME. Dynamic start and stop. > > > > 3) In theory, should cancel them and new setting should be applied. > How > > > it > > > > works now? Create an index and stop node, for example. > > > > > > > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov < > voze...@gridgain.com> > > > > wrote: > > > > > > > > > Hi Ed, > > > > > > > > > > Several questions from my side: > > > > > 1) If we do not allow to change the most demanded by users things > > like > > > > > affinity or persistence/in-memory, then what kind of configuration > > > > > properties do we expect to be changed? Can we have several > examples? > > > > > 2) How will it interact with PME? > > > > > 3) How will it interact with CREATE INDEX and ALTER TABLE commands? > > > > > > > > > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < > > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > > > Igniters, > > > > > > > > > > > > I propose new public API to change the cache configuration of > > > > persistent > > > > > > caches with keeping data. > > > > > > > > > > > > It would look like: > > > > > > > > > > > > Ignite ignite = ...; > > > > > > ignite.restartCaches(cfg1, ... cfgN); > > > > > > > > > > > > where cfgX is a new cache configuration, which contains the name > of > > > an > > > > > > existing persistent cache. > > > > > > > > > > >
[GitHub] zzzadruga opened a new pull request #78: IGNITE-10203 Support for alternative configurations for PR testing
zzzadruga opened a new pull request #78: IGNITE-10203 Support for alternative configurations for PR testing URL: https://github.com/apache/ignite-teamcity-bot/pull/78 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: New API for changing configuration of persistent caches
Eduard, Simple != correct. Let’s consider a simple use case: user want to change PARTITIONED -> REPLICATED from .NET, but do not some classes from CacheConfiguration. How do we solve this? Vladimir. ср, 21 нояб. 2018 г. в 18:02, Eduard Shangareev : > Vladimir, > > I propose not to change cache configuration in runtime but restart cache > with the new compatible configuration on data which we have underfoot. > > What we could change: > -backup count; > -TRANSACTIONAL <-> ATOMIC; > -REPLICATED - PARTITIONED; > -other settings. > > So, yeah, it would be great to have a possibility to change some properties > in runtime. But right we don't any way to change anything except indexes > and SQL fields. > > We already have all mechanism to do this. > The main issue is to make it reliable and exclude cases when we could come > to the unrecoverable state. > > So, I suggest keeping the solution as simple as possible. > For indexes clashes and ClassNotFoundException we could revert > configuration update and start with the old configuration. > > > On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov > wrote: > > > Eduard, > > > > Got it. Please take the following things in count during design: > > > > 1) Two distinct PMEs might not work well. Consider a situation w1hen I > > decided to move a cache with index "MY_INDEX" from schema A to schema B. > > While cache was stopped, another cache with index "MY_INDEX" was created > in > > schema B. Now first cache cannot start due to index name conflict. > > 2) Cancelling index creation is also bad idea because this is potentially > > long operation. Instead, most likely that we should wait to concurrent > > schema operations to finish first. That is, all operations on cache > should > > be ordered wrt each other somehow > > 3) Why do we think that cache restart will be needed at all? We have a > lot > > of configuration properties which could be changed safely either without > > PME or with a single PME. - rebalance properties, cache store properties > > (especially write-behind stuff), some query properties (e.g. "detail > > metrics"), etc.. In essence, it seems that >50% of properties could be > > changed without cache restart, other 25% will not be supported, and the > > rest may require restart. > > 4) Client nodes and thin client may not have necessary classes in > > classpath. E.g. consider a user which want to change rebalance timeout > for > > cache, but do not have configured interceptor in classpath. With proposed > > API it will be impossible. This is especially true for non-Java clients. > > > > That said, I think we should consider another API which will not require > > full CacheConfiguration object. This might be a kind of builder or so. > And > > once user set properties he want to change to the builder, we can analyze > > them and either change them in runtime without PME, change with a single > > PME or change with full cache restart. > > > > What do you think? > > > > Vladimir. > > > > On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev < > > eshangar...@gridgain.com> > > wrote: > > > > > Vladimir, > > > > > > 1) Affinity could be changed, but count of partition couldn't be. > > > 2) So it would trigger 2 PME. Dynamic start and stop. > > > 3) In theory, should cancel them and new setting should be applied. How > > it > > > works now? Create an index and stop node, for example. > > > > > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov > > > wrote: > > > > > > > Hi Ed, > > > > > > > > Several questions from my side: > > > > 1) If we do not allow to change the most demanded by users things > like > > > > affinity or persistence/in-memory, then what kind of configuration > > > > properties do we expect to be changed? Can we have several examples? > > > > 2) How will it interact with PME? > > > > 3) How will it interact with CREATE INDEX and ALTER TABLE commands? > > > > > > > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > Igniters, > > > > > > > > > > I propose new public API to change the cache configuration of > > > persistent > > > > > caches with keeping data. > > > > > > > > > > It would look like: > > > > > > > > > > Ignite ignite = ...; > > > > > ignite.restartCaches(cfg1, ... cfgN); > > > > > > > > > > where cfgX is a new cache configuration, which contains the name of > > an > > > > > existing persistent cache. > > > > > > > > > > The obvious limitation: > > > > > - affinity key mapping couldn't be changed; > > > > > - count of partitions couldn't be changed; > > > > > - MVCC couldn't be turned off/on; > > > > > - persistent couldn't be turned off; > > > > > - group settings couldn't be changed (group name); > > > > > - if cache belongs to group it's needed to restart all of them. > > > > > > > > > > Failure scenario is the crucial thing (and most difficult): > > > > > - initiator fail; > > > > > - cluster restart at any stage; > > > > > - joining/starting offline nodes. > > > > > >
[GitHub] dspavlov opened a new pull request #77: IGNITE-10336: Build chain collection refactored to avoid missing builds
dspavlov opened a new pull request #77: IGNITE-10336: Build chain collection refactored to avoid missing builds URL: https://github.com/apache/ignite-teamcity-bot/pull/77 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: New API for changing configuration of persistent caches
Vladimir, I propose not to change cache configuration in runtime but restart cache with the new compatible configuration on data which we have underfoot. What we could change: -backup count; -TRANSACTIONAL <-> ATOMIC; -REPLICATED - PARTITIONED; -other settings. So, yeah, it would be great to have a possibility to change some properties in runtime. But right we don't any way to change anything except indexes and SQL fields. We already have all mechanism to do this. The main issue is to make it reliable and exclude cases when we could come to the unrecoverable state. So, I suggest keeping the solution as simple as possible. For indexes clashes and ClassNotFoundException we could revert configuration update and start with the old configuration. On Wed, Nov 21, 2018 at 5:44 PM Vladimir Ozerov wrote: > Eduard, > > Got it. Please take the following things in count during design: > > 1) Two distinct PMEs might not work well. Consider a situation w1hen I > decided to move a cache with index "MY_INDEX" from schema A to schema B. > While cache was stopped, another cache with index "MY_INDEX" was created in > schema B. Now first cache cannot start due to index name conflict. > 2) Cancelling index creation is also bad idea because this is potentially > long operation. Instead, most likely that we should wait to concurrent > schema operations to finish first. That is, all operations on cache should > be ordered wrt each other somehow > 3) Why do we think that cache restart will be needed at all? We have a lot > of configuration properties which could be changed safely either without > PME or with a single PME. - rebalance properties, cache store properties > (especially write-behind stuff), some query properties (e.g. "detail > metrics"), etc.. In essence, it seems that >50% of properties could be > changed without cache restart, other 25% will not be supported, and the > rest may require restart. > 4) Client nodes and thin client may not have necessary classes in > classpath. E.g. consider a user which want to change rebalance timeout for > cache, but do not have configured interceptor in classpath. With proposed > API it will be impossible. This is especially true for non-Java clients. > > That said, I think we should consider another API which will not require > full CacheConfiguration object. This might be a kind of builder or so. And > once user set properties he want to change to the builder, we can analyze > them and either change them in runtime without PME, change with a single > PME or change with full cache restart. > > What do you think? > > Vladimir. > > On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev < > eshangar...@gridgain.com> > wrote: > > > Vladimir, > > > > 1) Affinity could be changed, but count of partition couldn't be. > > 2) So it would trigger 2 PME. Dynamic start and stop. > > 3) In theory, should cancel them and new setting should be applied. How > it > > works now? Create an index and stop node, for example. > > > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov > > wrote: > > > > > Hi Ed, > > > > > > Several questions from my side: > > > 1) If we do not allow to change the most demanded by users things like > > > affinity or persistence/in-memory, then what kind of configuration > > > properties do we expect to be changed? Can we have several examples? > > > 2) How will it interact with PME? > > > 3) How will it interact with CREATE INDEX and ALTER TABLE commands? > > > > > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < > > > eduard.shangar...@gmail.com> wrote: > > > > > > > Igniters, > > > > > > > > I propose new public API to change the cache configuration of > > persistent > > > > caches with keeping data. > > > > > > > > It would look like: > > > > > > > > Ignite ignite = ...; > > > > ignite.restartCaches(cfg1, ... cfgN); > > > > > > > > where cfgX is a new cache configuration, which contains the name of > an > > > > existing persistent cache. > > > > > > > > The obvious limitation: > > > > - affinity key mapping couldn't be changed; > > > > - count of partitions couldn't be changed; > > > > - MVCC couldn't be turned off/on; > > > > - persistent couldn't be turned off; > > > > - group settings couldn't be changed (group name); > > > > - if cache belongs to group it's needed to restart all of them. > > > > > > > > Failure scenario is the crucial thing (and most difficult): > > > > - initiator fail; > > > > - cluster restart at any stage; > > > > - joining/starting offline nodes. > > > > > > > > Some thoughts about implementation: > > > > - stop cache with destroy=false; > > > > - start cache dynamically with new configuration; > > > > - if indexes settings changed - remove index.bin to start indexation; > > > > - change blt-history when start cache initiated to not allow join > nodes > > > > with old configuration; > > > > - use restartId (IGNITE-8911) to not allow to start cache in between. > > > > > > > > Your thoughts? Would it be a useful feature? > > > > > > > > >
MVCC test coverage.
Hi Igniters, As you may already know, MVCC transaction feature will be available in upcoming Ignite-2.7. However, MVCC Tx feature is released for beta testing and has many limitations and we a going to improve stability and integrations with other features in future releases. We can reuse existed transactional cache tests and run them in Mvcc mode to get much better test coverage with small looses. Here is a ticket for this IGNITE-10001 [1]. This mean we will have twice more "Cache Tests" and get TC runs some longer. To reduce new Mvcc cache suites impact and save TC time we are going to 1. Include new tests to nightly suite only, this will allow us to put our ears to the ground. 2. Exclude non-tx tests and non-relevant tx cases and aggressively mute tests for unsupported features integrations. I've implement a PR to one of tasks [2] as an example how it can be done. Technical details: 1. Introduced a new FORCE_MVCC flag and created a child "Mvcc Cache 2" suite for "Cache 2" test suite with FORCE_MVCC flag on. 2. Implemented a hook that change TRANSACTIONAL cache atomicity mode to TRANSACTIONAL_SNAPSHOT if FORCE_MVCC flag turned on. This allow us to check MVCC mode without creating new test classes and minimal intervention in existed tests code. 3. All irrelevant tests marked as ignored in Mvcc suite. 4. Known unsupported cases are muted. New failures muted as well (corresponding tickets were created). Any pros\cons? Can someone please review a PR? [1] https://issues.apache.org/jira/browse/IGNITE-10001 [2] https://issues.apache.org/jira/browse/IGNITE-10002 -- Best regards, Andrey V. Mashenkov
Re: New API for changing configuration of persistent caches
Eduard, Got it. Please take the following things in count during design: 1) Two distinct PMEs might not work well. Consider a situation w1hen I decided to move a cache with index "MY_INDEX" from schema A to schema B. While cache was stopped, another cache with index "MY_INDEX" was created in schema B. Now first cache cannot start due to index name conflict. 2) Cancelling index creation is also bad idea because this is potentially long operation. Instead, most likely that we should wait to concurrent schema operations to finish first. That is, all operations on cache should be ordered wrt each other somehow 3) Why do we think that cache restart will be needed at all? We have a lot of configuration properties which could be changed safely either without PME or with a single PME. - rebalance properties, cache store properties (especially write-behind stuff), some query properties (e.g. "detail metrics"), etc.. In essence, it seems that >50% of properties could be changed without cache restart, other 25% will not be supported, and the rest may require restart. 4) Client nodes and thin client may not have necessary classes in classpath. E.g. consider a user which want to change rebalance timeout for cache, but do not have configured interceptor in classpath. With proposed API it will be impossible. This is especially true for non-Java clients. That said, I think we should consider another API which will not require full CacheConfiguration object. This might be a kind of builder or so. And once user set properties he want to change to the builder, we can analyze them and either change them in runtime without PME, change with a single PME or change with full cache restart. What do you think? Vladimir. On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev wrote: > Vladimir, > > 1) Affinity could be changed, but count of partition couldn't be. > 2) So it would trigger 2 PME. Dynamic start and stop. > 3) In theory, should cancel them and new setting should be applied. How it > works now? Create an index and stop node, for example. > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov > wrote: > > > Hi Ed, > > > > Several questions from my side: > > 1) If we do not allow to change the most demanded by users things like > > affinity or persistence/in-memory, then what kind of configuration > > properties do we expect to be changed? Can we have several examples? > > 2) How will it interact with PME? > > 3) How will it interact with CREATE INDEX and ALTER TABLE commands? > > > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < > > eduard.shangar...@gmail.com> wrote: > > > > > Igniters, > > > > > > I propose new public API to change the cache configuration of > persistent > > > caches with keeping data. > > > > > > It would look like: > > > > > > Ignite ignite = ...; > > > ignite.restartCaches(cfg1, ... cfgN); > > > > > > where cfgX is a new cache configuration, which contains the name of an > > > existing persistent cache. > > > > > > The obvious limitation: > > > - affinity key mapping couldn't be changed; > > > - count of partitions couldn't be changed; > > > - MVCC couldn't be turned off/on; > > > - persistent couldn't be turned off; > > > - group settings couldn't be changed (group name); > > > - if cache belongs to group it's needed to restart all of them. > > > > > > Failure scenario is the crucial thing (and most difficult): > > > - initiator fail; > > > - cluster restart at any stage; > > > - joining/starting offline nodes. > > > > > > Some thoughts about implementation: > > > - stop cache with destroy=false; > > > - start cache dynamically with new configuration; > > > - if indexes settings changed - remove index.bin to start indexation; > > > - change blt-history when start cache initiated to not allow join nodes > > > with old configuration; > > > - use restartId (IGNITE-8911) to not allow to start cache in between. > > > > > > Your thoughts? Would it be a useful feature? > > > > > > > > -- > Best regards, > Eduard. >
Re: [IMPORTANT] Future of Binary Objects
Possibly heading into wishlist rather than practical territory here, but you did ask... > What we need is to introduce an interface which will convert a pair of > key-value objects into a row. This row will be used to store data and to > get fields from it. Rather than mapping objects to a row, how about mapping to a more general “internal storage” interface? Assuming that all the data for a row is stored together makes it difficult to implement any optimisations that spans multiple rows. Think of a string state field where there are only five known values… we currently repeat the text over and over. Or a full column store backend. Regards, Stephen
Re: New API for changing configuration of persistent caches
Sure. thanks for the reply. I think in this case we should mention a new method in all error messages like 'cache create failed because of different configs'. So a user can easily find a new method to apply. So product will point itself to a method how to get the evolution of cache configs without re-loading all data. ср, 21 нояб. 2018 г. в 17:05, Eduard Shangareev : > Dmitriy, > > From my point of view, it's not good to have such unexpected behavior (and > change old one) for cache and getOrCreateCache method. > It should be a conscious decision, not an accident. > > On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev < > eshangar...@gridgain.com> > wrote: > > > Vladimir, > > > > 1) Affinity could be changed, but count of partition couldn't be. > > 2) So it would trigger 2 PME. Dynamic start and stop. > > 3) In theory, should cancel them and new setting should be applied. How > it > > works now? Create an index and stop node, for example. > > > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov > > wrote: > > > > > Hi Ed, > > > > > > Several questions from my side: > > > 1) If we do not allow to change the most demanded by users things like > > > affinity or persistence/in-memory, then what kind of configuration > > > properties do we expect to be changed? Can we have several examples? > > > 2) How will it interact with PME? > > > 3) How will it interact with CREATE INDEX and ALTER TABLE commands? > > > > > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < > > > eduard.shangar...@gmail.com> wrote: > > > > > > > Igniters, > > > > > > > > I propose new public API to change the cache configuration of > > persistent > > > > caches with keeping data. > > > > > > > > It would look like: > > > > > > > > Ignite ignite = ...; > > > > ignite.restartCaches(cfg1, ... cfgN); > > > > > > > > where cfgX is a new cache configuration, which contains the name of > an > > > > existing persistent cache. > > > > > > > > The obvious limitation: > > > > - affinity key mapping couldn't be changed; > > > > - count of partitions couldn't be changed; > > > > - MVCC couldn't be turned off/on; > > > > - persistent couldn't be turned off; > > > > - group settings couldn't be changed (group name); > > > > - if cache belongs to group it's needed to restart all of them. > > > > > > > > Failure scenario is the crucial thing (and most difficult): > > > > - initiator fail; > > > > - cluster restart at any stage; > > > > - joining/starting offline nodes. > > > > > > > > Some thoughts about implementation: > > > > - stop cache with destroy=false; > > > > - start cache dynamically with new configuration; > > > > - if indexes settings changed - remove index.bin to start indexation; > > > > - change blt-history when start cache initiated to not allow join > nodes > > > > with old configuration; > > > > - use restartId (IGNITE-8911) to not allow to start cache in between. > > > > > > > > Your thoughts? Would it be a useful feature? > > > > > > > > > > > > > -- > > Best regards, > > Eduard. > > >
Re: [IMPORTANT] Future of Binary Objects
Makes sense. I'm trying to grasp this from usability POV. Having two ways of storing data with different behavior can be confusing. In .NET we already have this issue with DateTime when if you want SQL, you get subtly different behavior. So IMO we should enable strict type checks for all caches, even non-SQL ones. Users will be able to evolve types by adding/removing fields, but at least type id will be fixed. And for SQL caches you'll get a clear exception like "Field does not exist in SQL schema: foobar" On Wed, Nov 21, 2018 at 4:19 PM Vladimir Ozerov wrote: > Pavel, > > This could be solved with aforementioned "RowFormat". We will be able to > configure cache as follows: "this is a cache with strict type checks, first > one is A, with fields A1, A2, A3, second is B with fields B1, B2". So it > will be possible to serialize anything into binary object, but when it > comes to real store, exception will be thrown. > > Makes sense? > > On Wed, Nov 21, 2018 at 3:21 PM Pavel Tupitsyn > wrote: > > > Vladimir, > > > > IMO the issue is that we allow any type of data in the cache (put Person, > > then put int to the same cache). > > Are we going to address this in 3.0 and enforce key/value types according > > to cache configuration? > > This will provide more space for optimizations. > > > > On Wed, Nov 21, 2018 at 3:14 PM Vladimir Ozerov > > wrote: > > > > > Denis, > > > > > > In theory data conversion could be avoided in certain cases. E.g. > > consider > > > a case of loading data through streamer. We know the cache, we know > it's > > > metadata and row format. So instead of doing "user object" -> "binary > > > object" -> "row", we can do "user object" -> "row". > > > > > > On Wed, Nov 21, 2018 at 1:31 PM Denis Mekhanikov < > dmekhani...@gmail.com> > > > wrote: > > > > > > > Vladimir, > > > > > > > > Thank you for the clarification. I didn't see this distinction first. > > > > > > > > I meant using customizable formats for all serialization, not only > for > > > > storage. > > > > The idea behind my proposal is to avoid data conversion, when loading > > > data > > > > into Ignite. > > > > It will complicate usage of thin clients though, so I'm not sure, > that > > it > > > > will make users happier. > > > > > > > > But anyway, the same approach may be used for storage only. > > > > > > > > Denis > > > > > > > > ср, 21 нояб. 2018 г. в 12:57, Vladimir Ozerov >: > > > > > > > > > Denis, > > > > > > > > > > Could you please clarify - are you talking about storage, e.g. how > > > > objects > > > > > are stored in Ignite, or about serialization as a whole? I'd like > to > > > > better > > > > > understand whether the use case you described is relevant to my > idea > > of > > > > > splitting binary objects from underlying storage format. > > > > > My vision was that we can use current BinaryObject protocol (with > > > > whatever > > > > > optimizations needed), as a common format for communication between > > > nodes > > > > > and a common serialization protocol. This is very handy because all > > > > > participants (Java, С++, .NET, all sorts of thin clients) are able > to > > > > work > > > > > with it. So if I have a "Person" class in Java I can read it in any > > > other > > > > > platform without any additional configuration. But when it comes to > > > > > *storage*, then we may introduce pluggable row format interface > which > > > > will > > > > > apply any necessary transformations. So if someone wants to store > > > objects > > > > > in Avro/Protobuf, and ready to configure and implement it (generate > > > > > classes, implementa field extraction logic, etc.) - then just > > implement > > > > > that interface. They key is that this implementation will only be > > > needed > > > > in > > > > > Java, not in a dozen of platform we support. > > > > > > > > > > But when it comes to how to store object in a cache > > > > > > > > > > On Wed, Nov 21, 2018 at 11:37 AM Denis Mekhanikov < > > > dmekhani...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > People often ask about possibility to store their data in that > > > format, > > > > > that > > > > > > they use in their applications. > > > > > > If you use Avro everywhere in your application, then why not > store > > > data > > > > > in > > > > > > the same format in Ignite? > > > > > > So, how about making an interface, that would enlist all > operations > > > we > > > > > > need, > > > > > > and use this interface everywhere without relying on any specific > > > > > > implementation. > > > > > > *BinaryObject* looks like a suitable interface, but the only > > > > > > implementation, that you can get from Ignite > > > > > > is *BinaryObjectImpl*. > > > > > > I think, we should make Ignite extendible and provide capability > to > > > > > specify > > > > > > your own data format > > > > > > by implementing the corresponding interfaces. > > > > > > So, if you like JSONB or Protobuf or whatever else, you could > > enable > > > a > > > > > > module f
Re: New API for changing configuration of persistent caches
Dmitriy, >From my point of view, it's not good to have such unexpected behavior (and change old one) for cache and getOrCreateCache method. It should be a conscious decision, not an accident. On Wed, Nov 21, 2018 at 5:01 PM Eduard Shangareev wrote: > Vladimir, > > 1) Affinity could be changed, but count of partition couldn't be. > 2) So it would trigger 2 PME. Dynamic start and stop. > 3) In theory, should cancel them and new setting should be applied. How it > works now? Create an index and stop node, for example. > > On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov > wrote: > > > Hi Ed, > > > > Several questions from my side: > > 1) If we do not allow to change the most demanded by users things like > > affinity or persistence/in-memory, then what kind of configuration > > properties do we expect to be changed? Can we have several examples? > > 2) How will it interact with PME? > > 3) How will it interact with CREATE INDEX and ALTER TABLE commands? > > > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < > > eduard.shangar...@gmail.com> wrote: > > > > > Igniters, > > > > > > I propose new public API to change the cache configuration of > persistent > > > caches with keeping data. > > > > > > It would look like: > > > > > > Ignite ignite = ...; > > > ignite.restartCaches(cfg1, ... cfgN); > > > > > > where cfgX is a new cache configuration, which contains the name of an > > > existing persistent cache. > > > > > > The obvious limitation: > > > - affinity key mapping couldn't be changed; > > > - count of partitions couldn't be changed; > > > - MVCC couldn't be turned off/on; > > > - persistent couldn't be turned off; > > > - group settings couldn't be changed (group name); > > > - if cache belongs to group it's needed to restart all of them. > > > > > > Failure scenario is the crucial thing (and most difficult): > > > - initiator fail; > > > - cluster restart at any stage; > > > - joining/starting offline nodes. > > > > > > Some thoughts about implementation: > > > - stop cache with destroy=false; > > > - start cache dynamically with new configuration; > > > - if indexes settings changed - remove index.bin to start indexation; > > > - change blt-history when start cache initiated to not allow join nodes > > > with old configuration; > > > - use restartId (IGNITE-8911) to not allow to start cache in between. > > > > > > Your thoughts? Would it be a useful feature? > > > > > > > > -- > Best regards, > Eduard. >
Re: New API for changing configuration of persistent caches
Vladimir, 1) Affinity could be changed, but count of partition couldn't be. 2) So it would trigger 2 PME. Dynamic start and stop. 3) In theory, should cancel them and new setting should be applied. How it works now? Create an index and stop node, for example. On Wed, Nov 21, 2018 at 4:56 PM Vladimir Ozerov wrote: > Hi Ed, > > Several questions from my side: > 1) If we do not allow to change the most demanded by users things like > affinity or persistence/in-memory, then what kind of configuration > properties do we expect to be changed? Can we have several examples? > 2) How will it interact with PME? > 3) How will it interact with CREATE INDEX and ALTER TABLE commands? > > On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < > eduard.shangar...@gmail.com> wrote: > > > Igniters, > > > > I propose new public API to change the cache configuration of persistent > > caches with keeping data. > > > > It would look like: > > > > Ignite ignite = ...; > > ignite.restartCaches(cfg1, ... cfgN); > > > > where cfgX is a new cache configuration, which contains the name of an > > existing persistent cache. > > > > The obvious limitation: > > - affinity key mapping couldn't be changed; > > - count of partitions couldn't be changed; > > - MVCC couldn't be turned off/on; > > - persistent couldn't be turned off; > > - group settings couldn't be changed (group name); > > - if cache belongs to group it's needed to restart all of them. > > > > Failure scenario is the crucial thing (and most difficult): > > - initiator fail; > > - cluster restart at any stage; > > - joining/starting offline nodes. > > > > Some thoughts about implementation: > > - stop cache with destroy=false; > > - start cache dynamically with new configuration; > > - if indexes settings changed - remove index.bin to start indexation; > > - change blt-history when start cache initiated to not allow join nodes > > with old configuration; > > - use restartId (IGNITE-8911) to not allow to start cache in between. > > > > Your thoughts? Would it be a useful feature? > > > -- Best regards, Eduard.
[GitHub] ignite pull request #5462: IGNITE-10369 PDS 4 hangs on TC
GitHub user ibessonov opened a pull request: https://github.com/apache/ignite/pull/5462 IGNITE-10369 PDS 4 hangs on TC You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10369 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5462.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5462 commit 4e042d7aff974fbe54af02d4ef76060f86791ce8 Author: ibessonov Date: 2018-11-21T13:58:39Z IGNITE-10369 Additional logging; Only one test left in suite so I can test it more effectively. ---
Re: New API for changing configuration of persistent caches
Hi Eduard, Can we just change cache parameters after calling cache() or getOrCreateCache() by user? The method is blocked until restart finished. We can set up new cache parameters according to the provided configuration. If it is not possible to change some parameter, a method will fail. I guess it would be the easiest way for users. So do we need a new method? Sincerely, Dmitriy Pavlov ср, 21 нояб. 2018 г. в 16:48, Eduard Shangareev : > Igniters, > > I propose new public API to change the cache configuration of persistent > caches with keeping data. > > It would look like: > > Ignite ignite = ...; > ignite.restartCaches(cfg1, ... cfgN); > > where cfgX is a new cache configuration, which contains the name of an > existing persistent cache. > > The obvious limitation: > - affinity key mapping couldn't be changed; > - count of partitions couldn't be changed; > - MVCC couldn't be turned off/on; > - persistent couldn't be turned off; > - group settings couldn't be changed (group name); > - if cache belongs to group it's needed to restart all of them. > > Failure scenario is the crucial thing (and most difficult): > - initiator fail; > - cluster restart at any stage; > - joining/starting offline nodes. > > Some thoughts about implementation: > - stop cache with destroy=false; > - start cache dynamically with new configuration; > - if indexes settings changed - remove index.bin to start indexation; > - change blt-history when start cache initiated to not allow join nodes > with old configuration; > - use restartId (IGNITE-8911) to not allow to start cache in between. > > Your thoughts? Would it be a useful feature? >
Re: New API for changing configuration of persistent caches
Hi Ed, Several questions from my side: 1) If we do not allow to change the most demanded by users things like affinity or persistence/in-memory, then what kind of configuration properties do we expect to be changed? Can we have several examples? 2) How will it interact with PME? 3) How will it interact with CREATE INDEX and ALTER TABLE commands? On Wed, Nov 21, 2018 at 4:48 PM Eduard Shangareev < eduard.shangar...@gmail.com> wrote: > Igniters, > > I propose new public API to change the cache configuration of persistent > caches with keeping data. > > It would look like: > > Ignite ignite = ...; > ignite.restartCaches(cfg1, ... cfgN); > > where cfgX is a new cache configuration, which contains the name of an > existing persistent cache. > > The obvious limitation: > - affinity key mapping couldn't be changed; > - count of partitions couldn't be changed; > - MVCC couldn't be turned off/on; > - persistent couldn't be turned off; > - group settings couldn't be changed (group name); > - if cache belongs to group it's needed to restart all of them. > > Failure scenario is the crucial thing (and most difficult): > - initiator fail; > - cluster restart at any stage; > - joining/starting offline nodes. > > Some thoughts about implementation: > - stop cache with destroy=false; > - start cache dynamically with new configuration; > - if indexes settings changed - remove index.bin to start indexation; > - change blt-history when start cache initiated to not allow join nodes > with old configuration; > - use restartId (IGNITE-8911) to not allow to start cache in between. > > Your thoughts? Would it be a useful feature? >
Re: Ignite web console on kubernetes with ingress
Hi Vishwas, Are you going to do some contribution related to your question? If not then you could ask the same question on u...@ignite.apache.org User list exists exactly for asking questions related to usability problems and most likely you will get an answer there. ср, 21 нояб. 2018 г. в 10:26, vbm : > > Hi, > > I was trying to use Ignite web console on Kubernetes using the docker image > at > > For external access, I am using ingress kubernetes resource. I am able to > access Web console UI using the default href path (/). > > But with non default href path, the ignite web console page does not load. > Is there any workaround for this ? > > Below is the ingress file, I am using: > > apiVersion: extensions/v1beta1 > kind: Ingress > metadata: > name: web-console-ingress > namespace: default > annotations: > ingress.kubernetes.io/rewrite-target: "/" > spec: > rules: >- http: >paths: > - backend: > serviceName: ignite-web-console-service > servicePort: 8080 > path: /ignite-web-console-ui > > > Regards, > Vishwas > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ -- Best regards, Ivan Pavlukhin
New API for changing configuration of persistent caches
Igniters, I propose new public API to change the cache configuration of persistent caches with keeping data. It would look like: Ignite ignite = ...; ignite.restartCaches(cfg1, ... cfgN); where cfgX is a new cache configuration, which contains the name of an existing persistent cache. The obvious limitation: - affinity key mapping couldn't be changed; - count of partitions couldn't be changed; - MVCC couldn't be turned off/on; - persistent couldn't be turned off; - group settings couldn't be changed (group name); - if cache belongs to group it's needed to restart all of them. Failure scenario is the crucial thing (and most difficult): - initiator fail; - cluster restart at any stage; - joining/starting offline nodes. Some thoughts about implementation: - stop cache with destroy=false; - start cache dynamically with new configuration; - if indexes settings changed - remove index.bin to start indexation; - change blt-history when start cache initiated to not allow join nodes with old configuration; - use restartId (IGNITE-8911) to not allow to start cache in between. Your thoughts? Would it be a useful feature?
Re: Improve lazy mode SQL query execution (IGNITE-9171)
Hi Taras, Unfortunately, this will not be easy to implement. We already have AffinityTopologyVersion which is updated on certain schema change operations, such as CREATE/DROP TABLE. Moreover, it is already passed in query request messages (GridH2QueryRequest.topVer). Unfortunately, it is not updated for other DDL scenarios, such as ALTER TABLE. We can introduce our own global counter. We can try to integrate with AffinityTopologyVersion. Both solutions will require tight integration with PME mechanism to function properly. But good news is that this is not a new problem. Currently it is already possible for two map requests to be executed on different schemas: client_1: SEND_QUERY client_2: SEND_ALTER node_1 :EXEC_QUERY EXEC_ALTER node_2 : EXEC_ALTER EXEC_QUERY So I would say that nothing changes for us even after proposed changes, and way may leave this problem aside for now. Makes sense? On Wed, Nov 21, 2018 at 4:15 PM Taras Ledkov wrote: > Vladimir, > > The query protocol will be changed in both solution. > > The tables versions must be added to the 'GridQueryNextPageResponse' > message > and must be compared on the reduce node too. Because the DLL / DML race > may be happens on different nodes. > > 21.11.2018 15:24, Vladimir Ozerov пишет: > > Taras, > > > > Thank you for analysis. I'd like to add that there is another solution - > > PME :-) In this case DDL will be blocked until current SELECTs and > updates > > are finished, and do not allow new operations to be executed. This > approach > > solves all issues - it avoids deadlocks between query threads and DDL > > threads when they are reordered on different nodes, it avoids starvation > of > > DDL operations, and it doesn't cancel any queries. But there is serious > > drawback - performance. The drawback is that it is more complex to > > implement (query protocol changes might be required), it blocks the > cluster > > even when it is needed, and it may destabilize PME mechanism, which is > > already on his last legs. > > > > For this reason killing queries which interleave with DDL looks like a > > balanced solution for now - it is reasonably simple, allows us to we > avoid > > OOME in many cases, and do not introduce any additional complexity for > > users, as they are already prepared for re-tries. > > > > But I would like to stress one thing - we will need integration with PME > at > > some point in time anyway. Some DDL operations are blocking in their > nature > > (e.g. DROP COLUMN). Other DDL operations may be non-blocking, but > blocking > > implementation may give them serious performance benefits (e.g. CREATE > > INDEX). > > > > So I propose to go with your solution for now, and start thinking about > SQL > > -> PME integration in the background. > > > > Vladimir. > > > > On Wed, Nov 21, 2018 at 2:23 PM Taras Ledkov > wrote: > > > >> Hi community, > >> > >> We will enhance lazy mode for SQL query execution. > >> > >> Lazy mode overview: > >> Lazy mode is related to H2 lazy mode when the all query results are not > >> copied to the RAM in some cases. > >> The mode it is applicable for SELECTs that doesn't not require > >> materialize all results in memory, e.g. simple scan plans, IDX lookup, > >> merge join etc. > >> And not applicable for SORT by not indexed fields, aggregates, nested > >> loops joins etc. > >> > >> When mode is applicable it produces result with iterator-like behavior > >> on server side and not consume RAM. > >> So the huge result set may be selected without OOME. > >> > >> The current implementation. > >> The current implementation is start separate thread for each query with > >> 'lazy=true'. > >> This is caused by the our implementation of 'GridH2Table'. In details: > >> the table's locks. > >> The table must be locked while result set is completed. > >> > >> When lazy is disabled a complete result is generated on the first step > >> of a query execution (then tables unlock) > >> and result is stored on the node and sent to other node (or client) page > >> by page. > >> > >> When lazy is enabled tables are locked until result set delivery to > client. > >> > >> The start new thread causes overhead for requests that returns small > >> result set. > >> But current table lock is used `ReentrantReadWriteLock` and we cannot > >> lock tables from one thread > >> of QUERY thread pool and unlock in the other thread (when query is > >> complete or cancel). > >> > >> The trivial solve is using the 'StampedLock' it solve the lock behavior, > >> but not solve the table DDL starvation / deadlock. > >> Example: > >> Lets the QUERY thread pool contains only one thread. The case is scaled > >> for any thread pool size. > >> Write operation that require to exclusive table lock is DDL operation. > >> > >> 1. The query Q0 acquires the shared lock for the table T, send first > >> page result and leave thread 'threadQP0' control. > >> 2. DDL0 blocks on write lock the table T at the 'threadWP0 '
Re: Service grid redesign
Hi Vyacheslav, Vladimir, Could you please invite me, if you will set up a call. ср, 21 нояб. 2018 г. в 13:08, Vladimir Ozerov : > Hi Vyacheslav, > > Still not clear enough for me. I do not see a reason to send another over a > ring in case of successful execution. The only reason is an error on a node > which require correction (re-deploy to other node, full service undeploy, > etc). > I think it makes sense to organize another call to discuss current > architecture. Otherwise we may spend too much time on emails. > > Vladimir. > > On Wed, Nov 21, 2018 at 12:57 PM Vyacheslav Daradur > wrote: > > > The full map is needed: > > 1) to propagate deployment results which could be different from > > locally calculated in case of any errors; > > 2) to transfer deployment errors across the cluster; > > 3) to undeploy exceeded the number of service instances if needed; > > 4) to get know other nodes that deployment process was finished, this > > need to avoid calling services which have not been deployed yet (or > > can't be deployed). We can't just store pending requests because of > > time to deploy one service instances which may be significant. > > On Wed, Nov 21, 2018 at 12:45 PM Vladimir Ozerov > > wrote: > > > > > > Vyacheslav, > > > > > > I looked at the document and failed to find explanation why full maps > are > > > needed. Could you point me to a place where it is explained? > > > I ask this because my impression from last discussion was that it is > > never > > > needed. Service status change is initiated by user action, then all > nodes > > > perform respective action locally, then they reply to coordinator, then > > > coordinator reply to the client, no need a kind of "full" map over > > > discovery again. The only situation when another message over ring is > > > required, is when some node failed to execute local operation (for > > whatever > > > reason) and corrective action is required. > > > > > > Am I missing something? > > > > > > On Wed, Nov 21, 2018 at 11:50 AM Vyacheslav Daradur < > daradu...@gmail.com > > > > > > wrote: > > > > > > > Denis, I suggested new names above in the thread. > > > > > > > > Please, look at PME document [1] is should be quiet actual to show > the > > > > same flow. > > > > > > > > [1] > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/%28Partition+Map%29+Exchange+-+under+the+hood > > > > > > > > On Wed, Nov 21, 2018 at 11:43 AM Denis Mekhanikov < > > dmekhani...@gmail.com> > > > > wrote: > > > > > > > > > > Vyacheslav, > > > > > > > > > > Actually, the service assignment is implemented in a way, > > > > > that allows every node calculate the assignment itself, so no > > information > > > > > needs to be shared. > > > > > The only data, that is sent between nodes is deployment results, > > > > > and I don't see an analogy with exchange here. > > > > > > > > > > Denis > > > > > > > > > > ср, 21 нояб. 2018 г. в 11:16, Vladimir Ozerov < > voze...@gridgain.com > > >: > > > > > > > > > > > Hi Vyacheslav, > > > > > > > > > > > > Could you please explain in what situation coordinator needs to > > collect > > > > > > service deployments info from all nodes and share it with the > > cluster? > > > > I > > > > > > cannot remember from our design discussion when it is needed. > > Global > > > > state > > > > > > normally shared through discovery and only on node join, In this > > case > > > > we > > > > > > use "DiscoveryDataBags", not separate messages. > > > > > > > > > > > > On Wed, Nov 21, 2018 at 11:11 AM Vyacheslav Daradur < > > > > daradu...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > I think request-response is not suitable terms. > > > > > > > > > > > > > > Nodes send to coordinator maps of actual service deployments > > which > > > > > > > contains what count of instances of each service node hosts > > locally. > > > > > > > > > > > > > > Coordinator sends to the cluster the full map of deployments > > across > > > > the > > > > > > > cluster. > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 11:04 AM Vladimir Ozerov < > > > > voze...@gridgain.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > I do not know what is correct term :-) What I said is that > > > > "exchange" > > > > > > is > > > > > > > > counter intuitive here. There is no "exchange", instead nodes > > send > > > > > > > > information to coordinator that they finished some operation. > > E.g. > > > > we > > > > > > do > > > > > > > > the same for schema changes (index creation), and as Denis > > > > suggested, > > > > > > > > Request-Response is correct suffixes here. Message name > should > > > > explain > > > > > > > what > > > > > > > > really happened, instead of describing things which are > > somewhat > > > > > > similar > > > > > > > in > > > > > > > > internal flow. > > > > > > > > > > > > > > > > On Wed, Nov 21, 2018 at 10:49 AM Nikolay Izhikov < > > > > nizhi...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello, Vladimir. > > > > >
Re: JdbcColumnMeta
Hi Peter,
I am not aware of such ticket.
On Wed, Nov 21, 2018 at 2:23 PM Peter Borissow
wrote:
> Hello Vladimir,Is there a ticket for the issue I described in JIRA or
> should we create one and mark it blocked pending #6173?
>
> Thanks,Peter
>
> On Wednesday, November 21, 2018, 3:30:21 AM EST, Vladimir Ozerov <
> voze...@gridgain.com> wrote:
>
> Hi Peter,
>
> We will be able to fix this in consistent way for both "thin" and "thick"
> JDBC drivers once IGNITE-6173 [1] is merged. In this ticket we ensure that
> metadata for all schema objects are present on all nodes, so that we can
> simply delegate metadata request to underlying H2 table.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-6173
>
> On Tue, Nov 20, 2018 at 8:06 PM Peter Borissow
> wrote:
>
> > Dear Devs,I would like to help expand the jdbc resultset metadata
> > available when querying a table. Consider the following example:
> >
> >
> > CREATE TABLE ARTICLE (
> >ID BIGINT,
> >BODY text,
> >LAST_MODIFIED TIMESTAMP with time zone,
> >CONSTRAINT PK_ ARTICLE PRIMARY KEY (ID)
> > );
> > After creating the table, I can query the column metadata using the
> > java.sql.ResultSetMetaData. Example:
> >
> > java.sql.ResultSet rs = stmt.executeQuery("select * from article);
> > java.sql.ResultSetMetaData rsmd = rs.getMetaData();
> >
> > However, the resultset /column metadata will return the following types:
> >
> > - id: long- body: object- last_modified: object
> > Note that the body and last_modified fields are identified as generic
> > objects instead of their formal java types. This is not consistent with
> > other jdbc drivers (e.g. postgresql, h2, etc).
> >
> > After a little digging, I found the
> JdbcColumnMeta(GridQueryFieldMetadata
> > info) constructor which calls the JdbcThinUtils class. However, at this
> > point the GridQueryFieldMetadata is already mapped to a generic "object"
> so
> > it is impossible to return the correct type from the JdbcThinUtils class.
> >
> > Next, I decided to see how tables are created in the first place.
> > Specifically, the GridSqlQueryParser.parseCreateTable() method. In
> there, I
> > can see the correct column metadata. For example, if I add the following
> > code in the parseCreateTable method...
> > for (Column col : data.columns){
> >System.out.println(col.getName() + " (" + col.getType() + ")");
> > }
> >
> > ...it will print the following:
> > ID (5)
> > BODY (16)
> > LAST_MODIFIED (24)
> >
> > I would like to preserve this metadata so that I can see it when using
> the
> > jdbc resultset metadata object. Any suggestions?
> >
> > Note that the ID column type (5) is preserved. I suspect the other types
> > identified in the JdbcThinUtils class are preserved as well.
> >
> >
> > Thanks in advance,Peter
> >
> >
> >
> >
> >
> >
> >
> >
>
Re: [IMPORTANT] Future of Binary Objects
Pavel, This could be solved with aforementioned "RowFormat". We will be able to configure cache as follows: "this is a cache with strict type checks, first one is A, with fields A1, A2, A3, second is B with fields B1, B2". So it will be possible to serialize anything into binary object, but when it comes to real store, exception will be thrown. Makes sense? On Wed, Nov 21, 2018 at 3:21 PM Pavel Tupitsyn wrote: > Vladimir, > > IMO the issue is that we allow any type of data in the cache (put Person, > then put int to the same cache). > Are we going to address this in 3.0 and enforce key/value types according > to cache configuration? > This will provide more space for optimizations. > > On Wed, Nov 21, 2018 at 3:14 PM Vladimir Ozerov > wrote: > > > Denis, > > > > In theory data conversion could be avoided in certain cases. E.g. > consider > > a case of loading data through streamer. We know the cache, we know it's > > metadata and row format. So instead of doing "user object" -> "binary > > object" -> "row", we can do "user object" -> "row". > > > > On Wed, Nov 21, 2018 at 1:31 PM Denis Mekhanikov > > wrote: > > > > > Vladimir, > > > > > > Thank you for the clarification. I didn't see this distinction first. > > > > > > I meant using customizable formats for all serialization, not only for > > > storage. > > > The idea behind my proposal is to avoid data conversion, when loading > > data > > > into Ignite. > > > It will complicate usage of thin clients though, so I'm not sure, that > it > > > will make users happier. > > > > > > But anyway, the same approach may be used for storage only. > > > > > > Denis > > > > > > ср, 21 нояб. 2018 г. в 12:57, Vladimir Ozerov : > > > > > > > Denis, > > > > > > > > Could you please clarify - are you talking about storage, e.g. how > > > objects > > > > are stored in Ignite, or about serialization as a whole? I'd like to > > > better > > > > understand whether the use case you described is relevant to my idea > of > > > > splitting binary objects from underlying storage format. > > > > My vision was that we can use current BinaryObject protocol (with > > > whatever > > > > optimizations needed), as a common format for communication between > > nodes > > > > and a common serialization protocol. This is very handy because all > > > > participants (Java, С++, .NET, all sorts of thin clients) are able to > > > work > > > > with it. So if I have a "Person" class in Java I can read it in any > > other > > > > platform without any additional configuration. But when it comes to > > > > *storage*, then we may introduce pluggable row format interface which > > > will > > > > apply any necessary transformations. So if someone wants to store > > objects > > > > in Avro/Protobuf, and ready to configure and implement it (generate > > > > classes, implementa field extraction logic, etc.) - then just > implement > > > > that interface. They key is that this implementation will only be > > needed > > > in > > > > Java, not in a dozen of platform we support. > > > > > > > > But when it comes to how to store object in a cache > > > > > > > > On Wed, Nov 21, 2018 at 11:37 AM Denis Mekhanikov < > > dmekhani...@gmail.com > > > > > > > > wrote: > > > > > > > > > People often ask about possibility to store their data in that > > format, > > > > that > > > > > they use in their applications. > > > > > If you use Avro everywhere in your application, then why not store > > data > > > > in > > > > > the same format in Ignite? > > > > > So, how about making an interface, that would enlist all operations > > we > > > > > need, > > > > > and use this interface everywhere without relying on any specific > > > > > implementation. > > > > > *BinaryObject* looks like a suitable interface, but the only > > > > > implementation, that you can get from Ignite > > > > > is *BinaryObjectImpl*. > > > > > I think, we should make Ignite extendible and provide capability to > > > > specify > > > > > your own data format > > > > > by implementing the corresponding interfaces. > > > > > So, if you like JSONB or Protobuf or whatever else, you could > enable > > a > > > > > module for the corresponding > > > > > format, and use it for storing the data. > > > > > > > > > > Denis > > > > > > > > > > ср, 21 нояб. 2018 г. в 10:10, Alexey Zinoviev < > > zaleslaw@gmail.com > > > >: > > > > > > > > > > > I'd like @Vyacheslav Daradur approach. > > > > > > > > > > > > Maybe somebody could have a look at UnsafeRow in Spark > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java > > > > > > UnsafeRow is a concrete InternalRow that represents a mutable > > > internal > > > > > > raw-memory (and hence unsafe) binary row format. > > > > > > > > > > > > P.S. If somebody is interested in this apporach, I could share > more > > > > > > information > > > > > > > > > > > > вт, 20 нояб. 2018 г. в 11:33, Se
Re: Improve lazy mode SQL query execution (IGNITE-9171)
Vladimir, The query protocol will be changed in both solution. The tables versions must be added to the 'GridQueryNextPageResponse' message and must be compared on the reduce node too. Because the DLL / DML race may be happens on different nodes. 21.11.2018 15:24, Vladimir Ozerov пишет: Taras, Thank you for analysis. I'd like to add that there is another solution - PME :-) In this case DDL will be blocked until current SELECTs and updates are finished, and do not allow new operations to be executed. This approach solves all issues - it avoids deadlocks between query threads and DDL threads when they are reordered on different nodes, it avoids starvation of DDL operations, and it doesn't cancel any queries. But there is serious drawback - performance. The drawback is that it is more complex to implement (query protocol changes might be required), it blocks the cluster even when it is needed, and it may destabilize PME mechanism, which is already on his last legs. For this reason killing queries which interleave with DDL looks like a balanced solution for now - it is reasonably simple, allows us to we avoid OOME in many cases, and do not introduce any additional complexity for users, as they are already prepared for re-tries. But I would like to stress one thing - we will need integration with PME at some point in time anyway. Some DDL operations are blocking in their nature (e.g. DROP COLUMN). Other DDL operations may be non-blocking, but blocking implementation may give them serious performance benefits (e.g. CREATE INDEX). So I propose to go with your solution for now, and start thinking about SQL -> PME integration in the background. Vladimir. On Wed, Nov 21, 2018 at 2:23 PM Taras Ledkov wrote: Hi community, We will enhance lazy mode for SQL query execution. Lazy mode overview: Lazy mode is related to H2 lazy mode when the all query results are not copied to the RAM in some cases. The mode it is applicable for SELECTs that doesn't not require materialize all results in memory, e.g. simple scan plans, IDX lookup, merge join etc. And not applicable for SORT by not indexed fields, aggregates, nested loops joins etc. When mode is applicable it produces result with iterator-like behavior on server side and not consume RAM. So the huge result set may be selected without OOME. The current implementation. The current implementation is start separate thread for each query with 'lazy=true'. This is caused by the our implementation of 'GridH2Table'. In details: the table's locks. The table must be locked while result set is completed. When lazy is disabled a complete result is generated on the first step of a query execution (then tables unlock) and result is stored on the node and sent to other node (or client) page by page. When lazy is enabled tables are locked until result set delivery to client. The start new thread causes overhead for requests that returns small result set. But current table lock is used `ReentrantReadWriteLock` and we cannot lock tables from one thread of QUERY thread pool and unlock in the other thread (when query is complete or cancel). The trivial solve is using the 'StampedLock' it solve the lock behavior, but not solve the table DDL starvation / deadlock. Example: Lets the QUERY thread pool contains only one thread. The case is scaled for any thread pool size. Write operation that require to exclusive table lock is DDL operation. 1. The query Q0 acquires the shared lock for the table T, send first page result and leave thread 'threadQP0' control. 2. DDL0 blocks on write lock the table T at the 'threadWP0 ' 3. The query Q1 blocks on read lock the 'threadQP0' (because the writer in the queue). The deadlock happens. Q0 never can finish and unlock because query pool hasn't free thread. The possible solution: 1. Don't use readlock at all. The lock is used only for write / exclusive (DDL) operations. 2. The DDL (exclusive) operation change the table version. 3. Each read operation (query execution, result page fetch) store the table version before start and compare with the table version on the end. If the version is changed the special retry exception is thrown. CONS: - The retry logic is less user-friendly. But the distributed SQL cannot protect the user from implement retry logic totally: e.g. cluster topology change must handled on user side by retry query implemented by user, because some data have been delivered to user and we don't track which data is delivered. PROS: - no deadlocks; - no contention on table lock for SQL query. What do you think? -- Taras Ledkov Mail-To: tled...@gridgain.com -- Taras Ledkov Mail-To: tled...@gridgain.com
[GitHub] ignite pull request #4812: IGNITE-9309 added runLocal
Github user oleg-ostanin closed the pull request at: https://github.com/apache/ignite/pull/4812 ---
[GitHub] ignite pull request #5461: IGNITE-10370: Fix TensorFlowLocalInferenceExample...
GitHub user dmitrievanthony opened a pull request: https://github.com/apache/ignite/pull/5461 IGNITE-10370: Fix TensorFlowLocalInferenceExample fail on Windows You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10370 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5461.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5461 commit 93b3748509774b5158919eb2a190db76af882824 Author: dmitrievanthony Date: 2018-11-21T12:52:51Z IGNITE-10370: Add try-with-resources to close streams in DirectorySerializer. commit 2e7d6f2af2fb0161e4745ee0015ee174ab9023ad Author: dmitrievanthony Date: 2018-11-21T12:55:01Z IGNITE-10370: Add try-with-resources to close streams in DirectorySerializer. ---
[jira] [Created] (IGNITE-10370) ML: TensorFlowLocalInferenceExample fails on Windows
Anton Dmitriev created IGNITE-10370:
---
Summary: ML: TensorFlowLocalInferenceExample fails on Windows
Key: IGNITE-10370
URL: https://issues.apache.org/jira/browse/IGNITE-10370
Project: Ignite
Issue Type: Bug
Components: ml
Affects Versions: 2.8
Reporter: Anton Dmitriev
Assignee: Anton Dmitriev
Fix For: 2.8
Example fails on Windows with the following trace:
```
2018-11-21 15:40:18.817134: I tensorflow/cc/saved_model/reader.cc:31] Reading
SavedModel from:
C:\Users\Gridgain\AppData\Local\Temp\tensorflow_saved_model_4023051695462839971
2018-11-21 15:40:18.827159: I tensorflow/cc/saved_model/reader.cc:54] Reading
meta graph with tags \{ serve }
2018-11-21 15:40:18.828577: I
tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports
instructions that this TensorFlow binary was not compiled to use: AVX2
2018-11-21 15:40:18.836707: I tensorflow/cc/saved_model/loader.cc:162]
Restoring SavedModel bundle.
2018-11-21 15:40:18.884983: I tensorflow/cc/saved_model/loader.cc:138] Running
MainOp with key saved_model_main_op on SavedModel bundle.
2018-11-21 15:40:18.892141: I tensorflow/cc/saved_model/loader.cc:259]
SavedModel load for tags \{ serve }; Status: success. Took 75041 microseconds.
Exception in thread "main" java.lang.RuntimeException:
java.nio.file.FileSystemException:
C:\Users\Gridgain\AppData\Local\Temp\tensorflow_saved_model_4023051695462839971\saved_model.pb:
...
at
org.apache.ignite.ml.inference.util.DirectorySerializer.deleteDirectory(DirectorySerializer.java:104)
at
org.apache.ignite.ml.inference.util.DirectorySerializer.deleteDirectory(DirectorySerializer.java:96)
at
org.apache.ignite.ml.inference.parser.TensorFlowSavedModelInfModelParser.parseModel(TensorFlowSavedModelInfModelParser.java:67)
at
org.apache.ignite.ml.inference.parser.TensorFlowBaseInfModelParser.parse(TensorFlowBaseInfModelParser.java:51)
at
org.apache.ignite.ml.inference.builder.SingleInfModelBuilder.build(SingleInfModelBuilder.java:32)
at
org.apache.ignite.examples.ml.inference.TensorFlowLocalInferenceExample.main(TensorFlowLocalInferenceExample.java:76)
Caused by: java.nio.file.FileSystemException:
C:\Users\Gridgain\AppData\Local\Temp\tensorflow_saved_model_4023051695462839971\saved_model.pb:
...
```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of visa caching.
SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of
visa caching.
URL: https://github.com/apache/ignite-teamcity-bot/pull/76#discussion_r235369867
##
File path:
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcbot/visa/TcBotTriggerAndSignOffService.java
##
@@ -299,6 +294,12 @@ public SimpleResult commentJiraEx(
if (!Strings.isNullOrEmpty(ticketFullName)) {
BuildsInfo buildsInfo = new BuildsInfo(srvId, prov,
ticketFullName, branchForTc);
+VisaRequest lastVisaReq =
visasHistoryStorage.getLastVisaRequest(buildsInfo.getContributionKey());
+
+if (Objects.nonNull(lastVisaReq) && lastVisaReq.isObserving())
+return new SimpleResult("Jira wasn't commented. \"Re-run
possible blockers & Comment JIRA\" was triggered for current branch." +
Review comment:
```suggestion
return new SimpleResult("Jira wasn't commented." +
+ " \"Re-run possible blockers & Comment JIRA\" was
triggered for current branch." +
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of visa caching.
SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of
visa caching.
URL: https://github.com/apache/ignite-teamcity-bot/pull/76#discussion_r235369173
##
File path:
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/db/TcHelperDb.java
##
@@ -93,6 +93,23 @@ public static void stop(Ignite ignite) {
Ignition.stop(ignite.name(), false);
}
+/** */
+@NotNull
+public static CacheConfiguration getCacheV3Config(String
name) {
+CacheConfiguration ccfg = new CacheConfiguration<>(name);
+
+ccfg.setAffinity(new RendezvousAffinityFunction(false, 8));
+
+return ccfg;
+}
+
+/** */
+public static CacheConfiguration getCacheV3TxConfig(String
name) {
+return TcHelperDb.getCacheV3Config(name).setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
+
Review comment:
```suggestion
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of visa caching.
SomeFire commented on a change in pull request #76: IGNITE-10275 Refactor of
visa caching.
URL: https://github.com/apache/ignite-teamcity-bot/pull/76#discussion_r235369350
##
File path:
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/db/TcHelperDb.java
##
@@ -93,6 +93,23 @@ public static void stop(Ignite ignite) {
Ignition.stop(ignite.name(), false);
}
+/** */
+@NotNull
+public static CacheConfiguration getCacheV3Config(String
name) {
+CacheConfiguration ccfg = new CacheConfiguration<>(name);
+
+ccfg.setAffinity(new RendezvousAffinityFunction(false, 8));
+
+return ccfg;
+}
+
+/** */
+public static CacheConfiguration getCacheV3TxConfig(String
name) {
+return TcHelperDb.getCacheV3Config(name).setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
+
+}
+
Review comment:
```suggestion
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (IGNITE-10369) PDS 4 hangs on TC
Ivan Bessonov created IGNITE-10369: -- Summary: PDS 4 hangs on TC Key: IGNITE-10369 URL: https://issues.apache.org/jira/browse/IGNITE-10369 Project: Ignite Issue Type: Test Reporter: Ivan Bessonov Assignee: Ivan Bessonov [https://ci.ignite.apache.org/viewLog.html?buildId=2365697&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_Pds4] org.apache.ignite.internal.processors.cache.IgniteClusterActivateDeactivateTestWithPersistenceAndMemoryReuse#testClientJoinsWhenActivationIsInProgress hangs on client connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Improve lazy mode SQL query execution (IGNITE-9171)
Taras, Thank you for analysis. I'd like to add that there is another solution - PME :-) In this case DDL will be blocked until current SELECTs and updates are finished, and do not allow new operations to be executed. This approach solves all issues - it avoids deadlocks between query threads and DDL threads when they are reordered on different nodes, it avoids starvation of DDL operations, and it doesn't cancel any queries. But there is serious drawback - performance. The drawback is that it is more complex to implement (query protocol changes might be required), it blocks the cluster even when it is needed, and it may destabilize PME mechanism, which is already on his last legs. For this reason killing queries which interleave with DDL looks like a balanced solution for now - it is reasonably simple, allows us to we avoid OOME in many cases, and do not introduce any additional complexity for users, as they are already prepared for re-tries. But I would like to stress one thing - we will need integration with PME at some point in time anyway. Some DDL operations are blocking in their nature (e.g. DROP COLUMN). Other DDL operations may be non-blocking, but blocking implementation may give them serious performance benefits (e.g. CREATE INDEX). So I propose to go with your solution for now, and start thinking about SQL -> PME integration in the background. Vladimir. On Wed, Nov 21, 2018 at 2:23 PM Taras Ledkov wrote: > Hi community, > > We will enhance lazy mode for SQL query execution. > > Lazy mode overview: > Lazy mode is related to H2 lazy mode when the all query results are not > copied to the RAM in some cases. > The mode it is applicable for SELECTs that doesn't not require > materialize all results in memory, e.g. simple scan plans, IDX lookup, > merge join etc. > And not applicable for SORT by not indexed fields, aggregates, nested > loops joins etc. > > When mode is applicable it produces result with iterator-like behavior > on server side and not consume RAM. > So the huge result set may be selected without OOME. > > The current implementation. > The current implementation is start separate thread for each query with > 'lazy=true'. > This is caused by the our implementation of 'GridH2Table'. In details: > the table's locks. > The table must be locked while result set is completed. > > When lazy is disabled a complete result is generated on the first step > of a query execution (then tables unlock) > and result is stored on the node and sent to other node (or client) page > by page. > > When lazy is enabled tables are locked until result set delivery to client. > > The start new thread causes overhead for requests that returns small > result set. > But current table lock is used `ReentrantReadWriteLock` and we cannot > lock tables from one thread > of QUERY thread pool and unlock in the other thread (when query is > complete or cancel). > > The trivial solve is using the 'StampedLock' it solve the lock behavior, > but not solve the table DDL starvation / deadlock. > Example: > Lets the QUERY thread pool contains only one thread. The case is scaled > for any thread pool size. > Write operation that require to exclusive table lock is DDL operation. > > 1. The query Q0 acquires the shared lock for the table T, send first > page result and leave thread 'threadQP0' control. > 2. DDL0 blocks on write lock the table T at the 'threadWP0 ' > 3. The query Q1 blocks on read lock the 'threadQP0' (because the writer > in the queue). > The deadlock happens. Q0 never can finish and unlock because query pool > hasn't free thread. > > The possible solution: > > 1. Don't use readlock at all. The lock is used only for write / > exclusive (DDL) operations. > 2. The DDL (exclusive) operation change the table version. > 3. Each read operation (query execution, result page fetch) store the > table version before start and compare with the table version on the > end. If the version is changed the special retry exception is thrown. > > CONS: > - The retry logic is less user-friendly. But the distributed SQL cannot > protect the user from implement retry logic totally: e.g. cluster > topology change must handled on user side by retry query implemented by > user, because some data have been delivered to user and we don't track > which data is delivered. > > PROS: > - no deadlocks; > - no contention on table lock for SQL query. > > What do you think? > > -- > > Taras Ledkov > Mail-To: tled...@gridgain.com > >
Re: Time to remove automated messages from the devlist?
Please start a vote according to https://www.apache.org/foundation/voting.html Anyone can start a vote, you don't need to be a PMC. I don't feel it is a very important issue to remove notifications from the list, as it can be easily filtered out using mail setup. But if someone feels it is really disturbing, please go ahead. I'm ok with GitHub redirection, but I will not drive this topic. ср, 21 нояб. 2018 г. в 11:40, Павлухин Иван : > Dmitriy, let's proceed with it. > вт, 20 нояб. 2018 г. в 23:20, Dmitriy Pavlov : > > > > One more thing I want to emphasize here. We can't just remove messages, > it > > _must_ be sent to some list, which is why we need some additional list, > > e.g. notifications@ for this. > > > > So only one option to proceed here is to run a formal vote on list > creation > > and redirection of github/gitbox messages to a new list. > > > > пн, 19 нояб. 2018 г. в 18:23, Dmitriy Pavlov : > > > > > Denis, we need because contributors do not announce their > > > intent/designs/etc manually. It is the best way ever? No, of course. > > > > > > We have consensus on PR removal, so let's do it and see results. > > > > > > пн, 19 нояб. 2018 г. в 18:11, Denis Mekhanikov >: > > > > > >> Dmitriy, > > >> > > >> If a person wants to track all new tickets, then he may go to JIRA, > create > > >> a filter for Ignite tickets > > >> and subscribe to it. JIRA has a pretty flexible configuration of > filters > > >> and subscriptions, so you can > > >> specify exactly what issues you are interested in, and how often you > want > > >> to receive these emails. > > >> This is much more convenient and more flexible than filtering emails > from > > >> a > > >> bot. > > >> > > >> So, most people ignore JIRA messages, and the ones who want to track > new > > >> tickets, > > >> may go to JIRA and configure their own filters. I don't see, why we > need > > >> to > > >> keep the forwarding to dev list. > > >> > > >> Denis > > >> > > >> пт, 16 нояб. 2018 г. в 22:30, Павлухин Иван : > > >> > > >> > Hi, > > >> > > > >> > Can we collect opinions about keeping messages of mentioned types on > > >> > dev list? From my side (+ means keeping on dev list): > > >> > TC bot + > > >> > Jira - > > >> > GitHub - > > >> > пт, 16 нояб. 2018 г. в 22:25, Dmitriy Pavlov : > > >> > > > > >> > > Importance is hardly definable and it is not possible that > importance > > >> is > > >> > > equal for everyone. You can say about other human emails it is not > > >> > > important if some product area is not interesting for you. So I > can > > >> only > > >> > > understand the terms: email needs action/does not need action. > > >> > > > > >> > > If some contributor never reacted to JIRA notification he or she > may > > >> > think > > >> > > it is not important. But even we have a majority of contributors > who > > >> > > ignores JIRA, it does not mean it is a right decision to switch it > > >> off. > > >> > We > > >> > > don't play in a democracy, hopefully. > > >> > > > > >> > > My suggestion now: keep showing an excellent example of > human-human > > >> > > interaction, announces, etc from all Ignite veterans (especially, > > >> PMCs), > > >> > so > > >> > > newcomers can use the same approach. > > >> > > > > >> > > If PRs removal to notifications@ will show a positive tendency in > > >> > > human-human interaction, I can easily agree with the second step. > Only > > >> > > practice is truth criteria. > > >> > > > > >> > > пт, 16 нояб. 2018 г. в 22:08, Vladimir Ozerov < > voze...@gridgain.com>: > > >> > > > > >> > > > We want important emails to be easily observable. This is the > only > > >> > goal. > > >> > > > > > >> > > > пт, 16 нояб. 2018 г. в 21:51, Dmitriy Pavlov < > dpav...@apache.org>: > > >> > > > > > >> > > > > I suggest to think in another paradigm, let's not classify > emails > > >> to > > >> > be > > >> > > > > automatically issued or not, lets separate emails to other > > >> classes: a > > >> > > > > needed action from humans or not needed. > > >> > > > > > > >> > > > > If you don't have any interest in a change announced by JIRA > issue > > >> > > > created > > >> > > > > email, you can just skip. If you can help with comments, > review, > > >> > etc, you > > >> > > > > can become watcher or comment ticket, you can also point to > > >> > duplicate. > > >> > > > > > > >> > > > > In that paradigm, > > >> > > > > A) PR is perfectly ok to be redirected to notifications@ .- > PR > > >> > creation > > >> > > > > does not require any action from anyone. > > >> > > > > B) JIRA - I'm not sure (maybe as a second step, if we will see > > >> > > > contributors > > >> > > > > will write about important tickets). And instead we can > discuss > > >> Open > > >> > -> > > >> > > > > Patch available transition, as a reviewer needed. > > >> > > > > C) TC Bot - I'm sure - should never be redirected. Hopefully, > it > > >> > will not > > >> > > > > generate any alerts. > > >> > > > > > > >> > > > > I hardly understand goal: is our target metric -
Re: [ANNOUNCE] Welcome Igor Seliverstov as a new committer
Congratulations, Igor! On Wed, Nov 21, 2018 at 10:11 AM Alexey Zinoviev wrote: > Great news, Igor, hope to use this great feature in ML module! > > вт, 20 нояб. 2018 г. в 11:08, Павлухин Иван : > > > Igor my congratulations! Keep going! > > > > 2018-11-20 9:01 GMT+01:00, Vladimir Ozerov : > > > The Apache Ignite Project Management Committee (PMC) has invited Igor > > > Seliverstov to become a new committer and are happy to announce that he > > has > > > accepted. > > > > > > Igor contributed a lot to "Transactional SQL" feature. > > > > > > Please join me in welcoming Igor and congratulating him on his new role > > in > > > the Apache Ignite Community. > > > > > > Good luck, Igor! > > > > > > Vladimir. > > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > > -- Best regards, Andrey V. Mashenkov
[jira] [Created] (IGNITE-10368) MVCC: Create "Cache 9" test suite for MVCC mode.
Andrew Mashenkov created IGNITE-10368: - Summary: MVCC: Create "Cache 9" test suite for MVCC mode. Key: IGNITE-10368 URL: https://issues.apache.org/jira/browse/IGNITE-10368 Project: Ignite Issue Type: Sub-task Components: mvcc Reporter: Andrew Mashenkov Create MVCC version of IgniteCacheTestSuite8 and add it to TC. All non-relevant tests should be marked as ignored. Failed tests should be muted and tickets should be created for unknown failure reasons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10367) MVCC: Create "Cache 8" test suite for MVCC mode.
Andrew Mashenkov created IGNITE-10367: - Summary: MVCC: Create "Cache 8" test suite for MVCC mode. Key: IGNITE-10367 URL: https://issues.apache.org/jira/browse/IGNITE-10367 Project: Ignite Issue Type: Sub-task Components: mvcc Reporter: Andrew Mashenkov Create MVCC version of IgniteCacheTestSuite and add it to TC. All non-relevant tests should be marked as ignored. Failed tests should be muted and tickets should be created for unknown failure reasons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [IMPORTANT] Future of Binary Objects
Vladimir, IMO the issue is that we allow any type of data in the cache (put Person, then put int to the same cache). Are we going to address this in 3.0 and enforce key/value types according to cache configuration? This will provide more space for optimizations. On Wed, Nov 21, 2018 at 3:14 PM Vladimir Ozerov wrote: > Denis, > > In theory data conversion could be avoided in certain cases. E.g. consider > a case of loading data through streamer. We know the cache, we know it's > metadata and row format. So instead of doing "user object" -> "binary > object" -> "row", we can do "user object" -> "row". > > On Wed, Nov 21, 2018 at 1:31 PM Denis Mekhanikov > wrote: > > > Vladimir, > > > > Thank you for the clarification. I didn't see this distinction first. > > > > I meant using customizable formats for all serialization, not only for > > storage. > > The idea behind my proposal is to avoid data conversion, when loading > data > > into Ignite. > > It will complicate usage of thin clients though, so I'm not sure, that it > > will make users happier. > > > > But anyway, the same approach may be used for storage only. > > > > Denis > > > > ср, 21 нояб. 2018 г. в 12:57, Vladimir Ozerov : > > > > > Denis, > > > > > > Could you please clarify - are you talking about storage, e.g. how > > objects > > > are stored in Ignite, or about serialization as a whole? I'd like to > > better > > > understand whether the use case you described is relevant to my idea of > > > splitting binary objects from underlying storage format. > > > My vision was that we can use current BinaryObject protocol (with > > whatever > > > optimizations needed), as a common format for communication between > nodes > > > and a common serialization protocol. This is very handy because all > > > participants (Java, С++, .NET, all sorts of thin clients) are able to > > work > > > with it. So if I have a "Person" class in Java I can read it in any > other > > > platform without any additional configuration. But when it comes to > > > *storage*, then we may introduce pluggable row format interface which > > will > > > apply any necessary transformations. So if someone wants to store > objects > > > in Avro/Protobuf, and ready to configure and implement it (generate > > > classes, implementa field extraction logic, etc.) - then just implement > > > that interface. They key is that this implementation will only be > needed > > in > > > Java, not in a dozen of platform we support. > > > > > > But when it comes to how to store object in a cache > > > > > > On Wed, Nov 21, 2018 at 11:37 AM Denis Mekhanikov < > dmekhani...@gmail.com > > > > > > wrote: > > > > > > > People often ask about possibility to store their data in that > format, > > > that > > > > they use in their applications. > > > > If you use Avro everywhere in your application, then why not store > data > > > in > > > > the same format in Ignite? > > > > So, how about making an interface, that would enlist all operations > we > > > > need, > > > > and use this interface everywhere without relying on any specific > > > > implementation. > > > > *BinaryObject* looks like a suitable interface, but the only > > > > implementation, that you can get from Ignite > > > > is *BinaryObjectImpl*. > > > > I think, we should make Ignite extendible and provide capability to > > > specify > > > > your own data format > > > > by implementing the corresponding interfaces. > > > > So, if you like JSONB or Protobuf or whatever else, you could enable > a > > > > module for the corresponding > > > > format, and use it for storing the data. > > > > > > > > Denis > > > > > > > > ср, 21 нояб. 2018 г. в 10:10, Alexey Zinoviev < > zaleslaw@gmail.com > > >: > > > > > > > > > I'd like @Vyacheslav Daradur approach. > > > > > > > > > > Maybe somebody could have a look at UnsafeRow in Spark > > > > > > > > > > > > > > > > > > > > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java > > > > > UnsafeRow is a concrete InternalRow that represents a mutable > > internal > > > > > raw-memory (and hence unsafe) binary row format. > > > > > > > > > > P.S. If somebody is interested in this apporach, I could share more > > > > > information > > > > > > > > > > вт, 20 нояб. 2018 г. в 11:33, Sergi Vladykin < > > sergi.vlady...@gmail.com > > > >: > > > > > > > > > > > I really like Protobuf format. It is probably not what we need > for > > > O(1) > > > > > > fields access, > > > > > > but for compact data representation we can derive lots from > there. > > > > > > > > > > > > Also IMO, restricting field type change is absolutely sane idea. > > > > > > The correct way to evolve schema in common case is to add new > > fields > > > > and > > > > > > gradually > > > > > > deprecate the old ones, if you can skip default/null fields in > > binary > > > > > > format this approach > > > > > > will not introduce any noticeable performance/size overhead. > > > > > > >
[GitHub] ignite pull request #5057: IGNITE-9928
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5057 ---
[jira] [Created] (IGNITE-10366) MVCC: Create "Cache 1" test suite for MVCC mode.
Andrew Mashenkov created IGNITE-10366: - Summary: MVCC: Create "Cache 1" test suite for MVCC mode. Key: IGNITE-10366 URL: https://issues.apache.org/jira/browse/IGNITE-10366 Project: Ignite Issue Type: Sub-task Components: mvcc Reporter: Andrew Mashenkov Create MVCC version of IgniteCacheTestSuite6 and add it to TC. All non-relevant tests should be marked as ignored. Failed tests should be muted and tickets should be created for unknown failure reasons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-10365) MVCC: Create "Cache 6" test suite for MVCC mode.
Andrew Mashenkov created IGNITE-10365: - Summary: MVCC: Create "Cache 6" test suite for MVCC mode. Key: IGNITE-10365 URL: https://issues.apache.org/jira/browse/IGNITE-10365 Project: Ignite Issue Type: Sub-task Components: mvcc Reporter: Andrew Mashenkov Create MVCC version of IgniteCacheTestSuite5 and add it to TC. All non-relevant tests should be marked as ignored. Failed tests should be muted and tickets should be created for unknown failure reasons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [IMPORTANT] Future of Binary Objects
Denis, In theory data conversion could be avoided in certain cases. E.g. consider a case of loading data through streamer. We know the cache, we know it's metadata and row format. So instead of doing "user object" -> "binary object" -> "row", we can do "user object" -> "row". On Wed, Nov 21, 2018 at 1:31 PM Denis Mekhanikov wrote: > Vladimir, > > Thank you for the clarification. I didn't see this distinction first. > > I meant using customizable formats for all serialization, not only for > storage. > The idea behind my proposal is to avoid data conversion, when loading data > into Ignite. > It will complicate usage of thin clients though, so I'm not sure, that it > will make users happier. > > But anyway, the same approach may be used for storage only. > > Denis > > ср, 21 нояб. 2018 г. в 12:57, Vladimir Ozerov : > > > Denis, > > > > Could you please clarify - are you talking about storage, e.g. how > objects > > are stored in Ignite, or about serialization as a whole? I'd like to > better > > understand whether the use case you described is relevant to my idea of > > splitting binary objects from underlying storage format. > > My vision was that we can use current BinaryObject protocol (with > whatever > > optimizations needed), as a common format for communication between nodes > > and a common serialization protocol. This is very handy because all > > participants (Java, С++, .NET, all sorts of thin clients) are able to > work > > with it. So if I have a "Person" class in Java I can read it in any other > > platform without any additional configuration. But when it comes to > > *storage*, then we may introduce pluggable row format interface which > will > > apply any necessary transformations. So if someone wants to store objects > > in Avro/Protobuf, and ready to configure and implement it (generate > > classes, implementa field extraction logic, etc.) - then just implement > > that interface. They key is that this implementation will only be needed > in > > Java, not in a dozen of platform we support. > > > > But when it comes to how to store object in a cache > > > > On Wed, Nov 21, 2018 at 11:37 AM Denis Mekhanikov > > > wrote: > > > > > People often ask about possibility to store their data in that format, > > that > > > they use in their applications. > > > If you use Avro everywhere in your application, then why not store data > > in > > > the same format in Ignite? > > > So, how about making an interface, that would enlist all operations we > > > need, > > > and use this interface everywhere without relying on any specific > > > implementation. > > > *BinaryObject* looks like a suitable interface, but the only > > > implementation, that you can get from Ignite > > > is *BinaryObjectImpl*. > > > I think, we should make Ignite extendible and provide capability to > > specify > > > your own data format > > > by implementing the corresponding interfaces. > > > So, if you like JSONB or Protobuf or whatever else, you could enable a > > > module for the corresponding > > > format, and use it for storing the data. > > > > > > Denis > > > > > > ср, 21 нояб. 2018 г. в 10:10, Alexey Zinoviev >: > > > > > > > I'd like @Vyacheslav Daradur approach. > > > > > > > > Maybe somebody could have a look at UnsafeRow in Spark > > > > > > > > > > > > > > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java > > > > UnsafeRow is a concrete InternalRow that represents a mutable > internal > > > > raw-memory (and hence unsafe) binary row format. > > > > > > > > P.S. If somebody is interested in this apporach, I could share more > > > > information > > > > > > > > вт, 20 нояб. 2018 г. в 11:33, Sergi Vladykin < > sergi.vlady...@gmail.com > > >: > > > > > > > > > I really like Protobuf format. It is probably not what we need for > > O(1) > > > > > fields access, > > > > > but for compact data representation we can derive lots from there. > > > > > > > > > > Also IMO, restricting field type change is absolutely sane idea. > > > > > The correct way to evolve schema in common case is to add new > fields > > > and > > > > > gradually > > > > > deprecate the old ones, if you can skip default/null fields in > binary > > > > > format this approach > > > > > will not introduce any noticeable performance/size overhead. > > > > > > > > > > Sergi > > > > > > > > > > вт, 20 нояб. 2018 г. в 11:12, Vyacheslav Daradur < > > daradu...@gmail.com > > > >: > > > > > > > > > > > I think, one of a possible way to reduce overhead and TCO - SQL > > > Scheme > > > > > > approach. > > > > > > > > > > > > That assumes that metadata will be stored separately from > > serialized > > > > > > data to reduce size. > > > > > > In this case, the most advantages of Binary Objects like access > in > > > > > > O(1) and access without deserialization may be achieved. > > > > > > On Tue, Nov 20, 2018 at 10:56 AM Vladimir Ozerov < > > > voze...@gridgain.com > > > > > > > > > > > wr
Re: proposed realization KILL QUERY command
Denis, Space is bad candidate because it is a whitespace. Without whitespaces we can have syntax without quotes at all. Any non-whitespace delimiter will work, though: KILL QUERY 45.1 KILL QUERY 45-1 KILL QUERY 45:1 On Wed, Nov 21, 2018 at 3:06 PM Юрий wrote: > Denis, > > Let's consider parameter of KILL QUERY just a string with some query id, > without any meaning for user. User just need to get the id and pass as > parameter to KILL QUERY command. > > Even if query is distributed it have single query id from user perspective > and will killed on all nodes. User just need to known one global query id. > > How it can works. > 1)SELECT * from running_queries > result is > query_id | node_id > | sql | schema_name | connection_id | duration > 123.33 | e0a69cb8-a1a8-45f6-b84d-ead367a0 | SELECT ... | ... > | 22 | 23456 > 333.31 | aaa6acb8-a4a5-42f6-f842-ead111b00020 | UPDATE... | ... > | 321| 346 > 2) KILL QUERY '123.33' > > So, user need select query_id from running_queries view and use it for KILL > QUERY command. > > I hope it became clearer. > > > > ср, 21 нояб. 2018 г. в 02:11, Denis Magda : > > > Folks, > > > > The decimal syntax is really odd - KILL QUERY > > '[node_order].[query_counter]' > > > > Confusing, let's use a space to separate parameters. > > > > Also, what if I want to halt a specific query with certain ID? Don't know > > the node number, just know that the query is distributed and runs across > > several machines. Sounds like the syntax still should consider > > [node_order/id] as an optional parameter. > > > > Probably, if you explain to me how an end user will use this command from > > the very beginning (how do I look for a query id and node id, etc) then > the > > things get clearer. > > > > -- > > Denis > > > > On Tue, Nov 20, 2018 at 1:03 AM Юрий > wrote: > > > > > Hi Vladimir, > > > > > > Thanks for your suggestion to use MANAGEMENT_POOL for processing > > > cancellation requests. > > > > > > About your questions. > > > 1) I'm going to implements SQL view to provide list of running queries. > > The > > > SQL VIEW has been a little bit discussed earlier. Proposed name is > > > *running_queries* with following columns: query_id, node_id, sql, > > > schema_name, connection_id, duration. Currently most of the information > > can > > > be retrieved through cache API, however it doesn't matter, any case we > > > need to expose SQL VIEW. Seem's you are right - the part should be > > > implemented firstly. > > > 2) Fully agree that we need to support all kind of SQL queries > > > (SLECT/DML/DDL, transactional, non transnational, local, distributed). > I > > > definitely sure that it will possible for all of above, however I'm not > > > sure about DDL - need to investigate it deeper. Also need to understand > > > that canceled DML operation can lead to partially updated data for non > > > transational caches. > > > > > > > > > > > > пн, 19 нояб. 2018 г. в 19:17, Vladimir Ozerov : > > > > > > > Hi Yuriy, > > > > > > > > I think we can use MANAGEMENT_POOL for this. It is already used for > > some > > > > internal Ignite tasks, and it appears to be a good candidate to > process > > > > cancel requests. > > > > > > > > But there are several things which are not clear enough for me at the > > > > moment: > > > > 1) How user is going to get the list of running queries in the first > > > place? > > > > Do we already have any SQL commands/views to get this information? > > > > 2) We need to ensure that KILL command will be processed properly by > > all > > > > kinds of SQL queries - SELECT/DML/DDL, non-transactional or > > > transactional, > > > > local queries and distributed queries. Will we be able to support all > > > these > > > > modes? > > > > > > > > Vladimir. > > > > > > > > On Mon, Nov 19, 2018 at 6:37 PM Юрий > > > wrote: > > > > > > > > > Hi Igniters, > > > > > > > > > > Earlier we agreed about syntax KILL QUERY > > > '[node_order].[query_counter]', > > > > > e.g. KILL QUERY '25.123' for single query or KILL QUERY '25.*' for > > all > > > > > queries on the node. Which is part of IEP-29 > > > > > < > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring > > > > > > > > > > > . > > > > > > > > > > Now I want to discuss internal realization of KILL query feature. > > > > > > > > > > My current vision is following: > > > > > After parsing, Ignite create KILL query command with two > parameters: > > > > > nodeOrderId, nodeQryId. To determine that need to kill all queries > > on a > > > > > node we can use negative value of query id, due to qry id always > have > > > > > positive values. > > > > > The command process at IgniteH2Indexing as native command. > > > > > By nodeOrderId we find node which initial for the query and send to > > the > > > > > node new GridQueryKillRequest with nodeQryId to TOPIC_QUERY wit
Re: proposed realization KILL QUERY command
Denis, Let's consider parameter of KILL QUERY just a string with some query id, without any meaning for user. User just need to get the id and pass as parameter to KILL QUERY command. Even if query is distributed it have single query id from user perspective and will killed on all nodes. User just need to known one global query id. How it can works. 1)SELECT * from running_queries result is query_id | node_id | sql | schema_name | connection_id | duration 123.33 | e0a69cb8-a1a8-45f6-b84d-ead367a0 | SELECT ... | ... | 22 | 23456 333.31 | aaa6acb8-a4a5-42f6-f842-ead111b00020 | UPDATE... | ... | 321| 346 2) KILL QUERY '123.33' So, user need select query_id from running_queries view and use it for KILL QUERY command. I hope it became clearer. ср, 21 нояб. 2018 г. в 02:11, Denis Magda : > Folks, > > The decimal syntax is really odd - KILL QUERY > '[node_order].[query_counter]' > > Confusing, let's use a space to separate parameters. > > Also, what if I want to halt a specific query with certain ID? Don't know > the node number, just know that the query is distributed and runs across > several machines. Sounds like the syntax still should consider > [node_order/id] as an optional parameter. > > Probably, if you explain to me how an end user will use this command from > the very beginning (how do I look for a query id and node id, etc) then the > things get clearer. > > -- > Denis > > On Tue, Nov 20, 2018 at 1:03 AM Юрий wrote: > > > Hi Vladimir, > > > > Thanks for your suggestion to use MANAGEMENT_POOL for processing > > cancellation requests. > > > > About your questions. > > 1) I'm going to implements SQL view to provide list of running queries. > The > > SQL VIEW has been a little bit discussed earlier. Proposed name is > > *running_queries* with following columns: query_id, node_id, sql, > > schema_name, connection_id, duration. Currently most of the information > can > > be retrieved through cache API, however it doesn't matter, any case we > > need to expose SQL VIEW. Seem's you are right - the part should be > > implemented firstly. > > 2) Fully agree that we need to support all kind of SQL queries > > (SLECT/DML/DDL, transactional, non transnational, local, distributed). I > > definitely sure that it will possible for all of above, however I'm not > > sure about DDL - need to investigate it deeper. Also need to understand > > that canceled DML operation can lead to partially updated data for non > > transational caches. > > > > > > > > пн, 19 нояб. 2018 г. в 19:17, Vladimir Ozerov : > > > > > Hi Yuriy, > > > > > > I think we can use MANAGEMENT_POOL for this. It is already used for > some > > > internal Ignite tasks, and it appears to be a good candidate to process > > > cancel requests. > > > > > > But there are several things which are not clear enough for me at the > > > moment: > > > 1) How user is going to get the list of running queries in the first > > place? > > > Do we already have any SQL commands/views to get this information? > > > 2) We need to ensure that KILL command will be processed properly by > all > > > kinds of SQL queries - SELECT/DML/DDL, non-transactional or > > transactional, > > > local queries and distributed queries. Will we be able to support all > > these > > > modes? > > > > > > Vladimir. > > > > > > On Mon, Nov 19, 2018 at 6:37 PM Юрий > > wrote: > > > > > > > Hi Igniters, > > > > > > > > Earlier we agreed about syntax KILL QUERY > > '[node_order].[query_counter]', > > > > e.g. KILL QUERY '25.123' for single query or KILL QUERY '25.*' for > all > > > > queries on the node. Which is part of IEP-29 > > > > < > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring > > > > > > > > > . > > > > > > > > Now I want to discuss internal realization of KILL query feature. > > > > > > > > My current vision is following: > > > > After parsing, Ignite create KILL query command with two parameters: > > > > nodeOrderId, nodeQryId. To determine that need to kill all queries > on a > > > > node we can use negative value of query id, due to qry id always have > > > > positive values. > > > > The command process at IgniteH2Indexing as native command. > > > > By nodeOrderId we find node which initial for the query and send to > the > > > > node new GridQueryKillRequest with nodeQryId to TOPIC_QUERY with not > > > QUERY > > > > POOL executor. > > > > At GridReduceQueryExecutor we add support of processing new > > > > GridQueryKillRequest > > > > which just run already exists cancelQueries method with given qryId > or > > > with > > > > all qryIds which currently running at the node in case at initial > KILL > > > > QUERY parameters used star symbol. > > > > > > > > I have a doubt which of thread pool we should use to process > > > > GridQueryKillRequest. > > > > My opinion it shouldn't be QUERY pool, due to the pool ca
[GitHub] ignite pull request #5460: IGNITE-10191 Incorrect comparison of lists in Ren...
GitHub user oignatenko opened a pull request: https://github.com/apache/ignite/pull/5460 IGNITE-10191 Incorrect comparison of lists in RendezvousAffinityFunctionSimpleBenchmark#testAffinityCompatibility You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10191 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5460.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5460 commit 1eeca908a8076a8317947dac8a46845964d1d7ea Author: Oleg Ignatenko Date: 2018-08-23T13:13:28Z IGNITE-9348 ML examples improvements - wip (logging improved) -- verified with diffs overview, executing the examples and studying execution logs commit e50b89c392568ba9b93935c4fa6c7f7f93f5ec6f Author: Oleg Ignatenko Date: 2018-08-23T14:45:57Z Revert "IGNITE-9348 ML examples improvements" This reverts commit 1eeca908a8076a8317947dac8a46845964d1d7ea. commit 474024b4c5bbdb3c0a4ed2f0a66238c8054c6674 Author: Oleg Ignatenko Date: 2018-08-23T14:57:34Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 9642b233b5701bdad47ebea163079160227c582a Author: Oleg Ignatenko Date: 2018-08-28T14:01:11Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 7fc16d013ab725d2ff2e1a1b042c983f11d0c4d4 Author: Oleg Ignatenko Date: 2018-08-28T15:13:02Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit d2caba67b156674f051f50faebeafe0871bf0914 Author: Oleg Ignatenko Date: 2018-08-29T13:14:07Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 16775dff51d71ea68b4a3dea98be552130c493ed Author: Oleg Ignatenko Date: 2018-08-30T09:00:56Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit aedb59929974fe205b949225c1a338c68c60cfc8 Author: Oleg Ignatenko Date: 2018-08-30T09:42:38Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 39c6482fcdca506aa33011ed21c98060b4a8c68b Author: Oleg Ignatenko Date: 2018-08-30T11:28:05Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 33b32a2760a6559c78283b927e3191180d8ed9e1 Author: Oleg Ignatenko Date: 2018-08-30T12:31:16Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 9531028ddd1aef9e95f7e8c8b528086739bbb1b0 Author: Oleg Ignatenko Date: 2018-08-30T14:06:34Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 28f22c6e2fffcb82717ba5da7be2cfd39715c4e3 Author: Oleg Ignatenko Date: 2018-08-30T16:41:51Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit aacac88db519187413b0fc5ff9d0e55b8f8cff22 Author: Oleg Ignatenko Date: 2018-08-31T10:12:32Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 897f920dde46022849b13f9fb86dba8e54119a56 Author: Oleg Ignatenko Date: 2018-08-31T13:57:14Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 114c79e54c1b316006ccc3ff22d20d902f9313df Author: Oleg Ignatenko Date: 2018-08-31T17:39:16Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit fd6b659bb8be1992618ce4ce91f568a0988b3afa Author: Oleg Ignatenko Date: 2018-09-02T06:11:42Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 6ae6d1d3cf9743d8d466be0330511ddc8589e944 Author: Oleg Ignatenko Date: 2018-09-03T10:27:35Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit e8b27dbd3d0c1cbdb7a7659175f5c7bb447482bf Author: Oleg Ignatenko Date: 2018-09-04T09:49:44Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 622c82efdd0aa182fadea6b7ffa5d4615521a3f5 Author: Oleg Ignatenko Date: 2018-09-05T10:50:28Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit fb844fe3751e2e8ae87e6b8030b0e4bd659df9c2 Author: Oleg Ignatenko Date: 2018-09-05T11:45:58Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 480ed78869277d7e32f725ab71bec9621f1ac03a Author: Oleg Ignatenko Date: 2018-09-06T07:52:55Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit c99762498f617c0e98ea3062a43c0b30092166ef Author: Oleg Ignatenko Date: 2018-09-06T14:45:04Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 2e17175225c72f747d370b5fee96f8be69d6d2cb Author: Oleg Ignatenko Date: 2018-09-06T17:33:54Z Merge branch 'master' of https://github.com/apache/ignite into master-ml commit 9ebcd9a2fe5966b0bf42a95484395867c15d863f Author: Oleg Ignatenko Date: 2018-09-07T13
[GitHub] ignite pull request #5459: IGNITE-8391
GitHub user vldpyatkov opened a pull request: https://github.com/apache/ignite/pull/5459 IGNITE-8391 Removing some WAL history segments leads to WAL rebalance hanging You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-8391 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5459.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5459 commit 3c83346f167f60fed1311883325d7bc3a596a7f0 Author: vd-pyatkov Date: 2018-11-21T11:50:00Z IGNITE-8391 Removing some WAL history segments leads to WAL rebalance hanging ---
Re: [ANNOUNCE] Welcome Pavel Kovalenko as a new committer
Congratulations! On Wed, Nov 21, 2018 at 12:30 PM Dmitriy Pavlov wrote: > Congrats, Pavel. Well deserved achievement. > > ср, 21 нояб. 2018 г., 12:09 Nikita Amelchev : > > > Congratulations, Pavel! > > ср, 21 нояб. 2018 г. в 11:47, Vyacheslav Daradur : > > > > > > Congrats Pavel, I'm looking forward to future PME optimizations! > > > On Wed, Nov 21, 2018 at 11:44 AM Alexey Goncharuk < > agoncha...@apache.org> > > wrote: > > > > > > > > The Apache Ignite Project Management Committee (PMC) has invited > Pavel > > > > Kovalenko to become a new committer and are happy to announce that he > > has > > > > accepted. > > > > > > > > Pavel was actively investigating and improving the speed of PME > during > > node > > > > join/leave scenarios and achieved great progress in these tasks. > > > > > > > > Please join me in welcoming Pavel and congratulating him on his new > > role in > > > > the Apache Ignite Community. > > > > > > > > Good luck, Pavel! > > > > > > > > --AG > > > > > > > > > > > > -- > > > Best Regards, Vyacheslav D. > > > > > > > > -- > > Best wishes, > > Amelchev Nikita > > > -- Best regards, Ilya
[GitHub] ignite pull request #5412: IGNITE-10260: MVCC: GetAndPutIfAbsent operation r...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5412 ---
[GitHub] ignite pull request #5458: IGNIE-10339 Fix connection to cluster failed when...
GitHub user ivandasch opened a pull request: https://github.com/apache/ignite/pull/5458 IGNIE-10339 Fix connection to cluster failed when invoking validate indexes closure from control.sh You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-10339 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/5458.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5458 commit 99183c3bee50f201b80caf2a0238e5ff789930d9 Author: Ivan Daschinskiy Date: 2018-11-20T12:22:49Z IGNIE-10339 Fix connection to cluster failed when invoking validate indexes closure from control.sh ---
Improve lazy mode SQL query execution (IGNITE-9171)
Hi community, We will enhance lazy mode for SQL query execution. Lazy mode overview: Lazy mode is related to H2 lazy mode when the all query results are not copied to the RAM in some cases. The mode it is applicable for SELECTs that doesn't not require materialize all results in memory, e.g. simple scan plans, IDX lookup, merge join etc. And not applicable for SORT by not indexed fields, aggregates, nested loops joins etc. When mode is applicable it produces result with iterator-like behavior on server side and not consume RAM. So the huge result set may be selected without OOME. The current implementation. The current implementation is start separate thread for each query with 'lazy=true'. This is caused by the our implementation of 'GridH2Table'. In details: the table's locks. The table must be locked while result set is completed. When lazy is disabled a complete result is generated on the first step of a query execution (then tables unlock) and result is stored on the node and sent to other node (or client) page by page. When lazy is enabled tables are locked until result set delivery to client. The start new thread causes overhead for requests that returns small result set. But current table lock is used `ReentrantReadWriteLock` and we cannot lock tables from one thread of QUERY thread pool and unlock in the other thread (when query is complete or cancel). The trivial solve is using the 'StampedLock' it solve the lock behavior, but not solve the table DDL starvation / deadlock. Example: Lets the QUERY thread pool contains only one thread. The case is scaled for any thread pool size. Write operation that require to exclusive table lock is DDL operation. 1. The query Q0 acquires the shared lock for the table T, send first page result and leave thread 'threadQP0' control. 2. DDL0 blocks on write lock the table T at the 'threadWP0 ' 3. The query Q1 blocks on read lock the 'threadQP0' (because the writer in the queue). The deadlock happens. Q0 never can finish and unlock because query pool hasn't free thread. The possible solution: 1. Don't use readlock at all. The lock is used only for write / exclusive (DDL) operations. 2. The DDL (exclusive) operation change the table version. 3. Each read operation (query execution, result page fetch) store the table version before start and compare with the table version on the end. If the version is changed the special retry exception is thrown. CONS: - The retry logic is less user-friendly. But the distributed SQL cannot protect the user from implement retry logic totally: e.g. cluster topology change must handled on user side by retry query implemented by user, because some data have been delivered to user and we don't track which data is delivered. PROS: - no deadlocks; - no contention on table lock for SQL query. What do you think? -- Taras Ledkov Mail-To: tled...@gridgain.com
Re: JdbcColumnMeta
Hello Vladimir, Is there a ticket for the issue I described in JIRA or
should we create one and mark it blocked pending #6173?
Thanks,Peter
On Wednesday, November 21, 2018, 3:30:21 AM EST, Vladimir Ozerov
wrote:
Hi Peter,
We will be able to fix this in consistent way for both "thin" and "thick"
JDBC drivers once IGNITE-6173 [1] is merged. In this ticket we ensure that
metadata for all schema objects are present on all nodes, so that we can
simply delegate metadata request to underlying H2 table.
[1] https://issues.apache.org/jira/browse/IGNITE-6173
On Tue, Nov 20, 2018 at 8:06 PM Peter Borissow
wrote:
> Dear Devs, I would like to help expand the jdbc resultset metadata
> available when querying a table. Consider the following example:
>
>
> CREATE TABLE ARTICLE (
> ID BIGINT,
> BODY text,
> LAST_MODIFIED TIMESTAMP with time zone,
> CONSTRAINT PK_ ARTICLE PRIMARY KEY (ID)
> );
> After creating the table, I can query the column metadata using the
> java.sql.ResultSetMetaData. Example:
>
> java.sql.ResultSet rs = stmt.executeQuery("select * from article);
> java.sql.ResultSetMetaData rsmd = rs.getMetaData();
>
> However, the resultset /column metadata will return the following types:
>
> - id: long- body: object- last_modified: object
> Note that the body and last_modified fields are identified as generic
> objects instead of their formal java types. This is not consistent with
> other jdbc drivers (e.g. postgresql, h2, etc).
>
> After a little digging, I found the JdbcColumnMeta(GridQueryFieldMetadata
> info) constructor which calls the JdbcThinUtils class. However, at this
> point the GridQueryFieldMetadata is already mapped to a generic "object" so
> it is impossible to return the correct type from the JdbcThinUtils class.
>
> Next, I decided to see how tables are created in the first place.
> Specifically, the GridSqlQueryParser.parseCreateTable() method. In there, I
> can see the correct column metadata. For example, if I add the following
> code in the parseCreateTable method...
> for (Column col : data.columns){
> System.out.println(col.getName() + " (" + col.getType() + ")");
> }
>
> ...it will print the following:
> ID (5)
> BODY (16)
> LAST_MODIFIED (24)
>
> I would like to preserve this metadata so that I can see it when using the
> jdbc resultset metadata object. Any suggestions?
>
> Note that the ID column type (5) is preserved. I suspect the other types
> identified in the JdbcThinUtils class are preserved as well.
>
>
> Thanks in advance,Peter
>
>
>
>
>
>
>
>
[jira] [Created] (IGNITE-10364) .Net: FailureHandlerTest should be reworked
Vyacheslav Koptilin created IGNITE-10364:
Summary: .Net: FailureHandlerTest should be reworked
Key: IGNITE-10364
URL: https://issues.apache.org/jira/browse/IGNITE-10364
Project: Ignite
Issue Type: Test
Components: platforms
Affects Versions: 2.7
Reporter: Vyacheslav Koptilin
Currently, {{FailureHandlerTest}} uses a broken {{CacheStoreFactory}} in order
to trigger the execution of {{FailureHandler}} when a cache with the given
factory is created.
https://issues.apache.org/jira/browse/IGNITE-8640 provides a fix that initiates
rollback procedure in that case and so, does not trigger {{FailureHandler}}
So, the test should use another approach to test {{FailureHandler. It seems
}}{{IoomFailureHandlerTest}} can be used as a starting point.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ignite pull request #5430: IGNITE-9872 [Test falied] IgniteSinkConnectorTest...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5430 ---
[GitHub] ignite pull request #5415: IGNITE-10234: Inference workflow for ML
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/5415 ---
[jira] [Created] (IGNITE-10363) Web console: unexpected unsaved changes confirmation
Pavel Konstantinov created IGNITE-10363: --- Summary: Web console: unexpected unsaved changes confirmation Key: IGNITE-10363 URL: https://issues.apache.org/jira/browse/IGNITE-10363 Project: Ignite Issue Type: Bug Reporter: Pavel Konstantinov Attachments: image-2018-11-21-17-33-17-545.png # create a new cluster # do not save # click Import form DB # go through all steps # Save !image-2018-11-21-17-33-17-545.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
