Re: [VOTE] Define Apache Ignite as a Distributed Database
+1 On Tue, Nov 24, 2020 at 3:25 AM Saikat Maitra wrote: > +1 > > On Mon, Nov 23, 2020 at 4:55 PM Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > +1 > > > > On Mon, Nov 23, 2020 at 2:44 PM Denis Magda wrote: > > > > > Igniters, > > > > > > With this vote, I'd like to formally wrap up our discussion on defining > > > Ignite as a "distributed database" instead of an "in-memory computing" > > > platform. See the following discussion for the rationale and context: > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Renaming-Ignite-s-product-category-td49246.html > > > > > > If the vote passes, the website will define Ignite as "a distributed > > > database for in-memory speed at petabyte scale" to underscore our > > in-memory > > > origins (and the primary reason why the project is selected) but not > > > diminishing our persistence capabilities. All prominent use cases such > as > > > caching, high-performance computing, etc. will remain visible on the > > > website. There is nothing wrong if a distributed database is used as a > > > cache or high-performance compute cluster; it's up to an application > > > developer to decide. > > > > > > Overall, please cast your vote for defining Ignite as a "distributed > > > database": > > > +1 - support the change > > > -1 - disagree with the change, explain why > > > 0 - neutral > > > > > > This is a majority vote that is open for the next 7 days and to be > closed > > > on Monday, Nov 30th: > > > https://www.timeanddate.com/countdown/to?year=2020=11=30 > > > > > > - > > > Denis > > > > > >
[jira] [Created] (IGNITE-13746) Document partition-aware data loading
Pavel Tupitsyn created IGNITE-13746:
---
Summary: Document partition-aware data loading
Key: IGNITE-13746
URL: https://issues.apache.org/jira/browse/IGNITE-13746
Project: Ignite
Issue Type: Improvement
Components: documentation
Reporter: Pavel Tupitsyn
Assignee: Denis A. Magda
Document the fact that CacheStore.loadCache discards non-primary entries, and
how to deal with that properly. Old documentation has this section:
https://apacheignite.readme.io/docs/data-loading#partition-aware-data-loading
And a callout:
{code}
In case of partitioned caches and 3rd party persistence such as a relational
database, keys that are not mapped to this node, either as primary or backups,
will be automatically discarded.
This is not relevant for Ignite Persistent Store where every node stores only
that data for which the node is either a primary or backup.
{code}
Additionally, it would be great to mention non-primary entries being discarded
in CacheStore Javadoc (and .NET XMLDoc):
*
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cache/store/CacheStore.java
*
https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/Apache.Ignite.Core/Cache/Store/ICacheStore.cs
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Ignite 3.0 development approach
Folks, I went ahead and created the repository [1]. I also configured a TeamCity project [2] that runs all available JUnit tests on every PR creation or update. It also sends the status update to GitHub so that it's reflected in the PR itself so that we can do merges directly from GitHub. Basic steps to make a change are described on the Wiki page [3]. Let me know if you have any questions. [1] https://github.com/apache/ignite-3 [2] https://ci.ignite.apache.org/project/ignite3 [3] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-DevelopmentProcess -Val On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Thanks, guys. It looks like we are much closer to the consensus now. I > totally on board with the plan, but I would also like to address the > short-term needs. As I've already mentioned earlier, there are several > active IEPs, but we still don't have even a preliminary technical process > for working on these IEPs. I believe this might be frustrating for the > folks who would like to commit code. > > The scope we agreed on is quite big, and it will surely take significant > time to implement all the changes and stabilize them. Therefore, it's clear > to me that we will have to maintain 2.x and 3.x in parallel for quite some > time - this needs to be addressed somehow. I'm convinced that having a > separate repo is the ONLY way to do that, and so far, I haven't heard any > clear alternatives or reasons why we shouldn't do this. > > That said, I'm inclined to proceed with this in the next few days - I will > create a repo and describe the process (which we, of course, can discuss > and modify going forward). Let's, at the very least, try and see where it > leads us. > > If someone has any concrete alternative options on how to we can maintain > two major versions in parallel, let's have another voice discussion this > Friday. If we do the meeting, we should set it up with a clear goal to make > a decision. Please let me know if there is interest in this. > > -Val > > On Mon, Nov 16, 2020 at 6:31 AM Alexey Goncharuk < > alexey.goncha...@gmail.com> wrote: > >> Good, >> >> I think we have an intermediate agreement on the scope and significance of >> the changes we want to make. I suggest creating separate discussion >> streams >> and calls for each of the suggested topics so that: >> >>- It is clear for the community what is the motivation of the stream >>(this includes both functional targets and technical debt issues >> pointed >>out by Sergey) >>- Who is planning to take an active part in each of the streams (i.e. >>the 'design committee', as Sergey suggested) >>- What are the intermediate and final goals for each of the streams >>- What are the cross-stream interactions and how we integrate them >>- How each of the streams will be integrated with the current codebase >>based on the above (here is where we will see whether drop-in or >>incremental approaches make more sense) >> >
[GitHub] [ignite-3] vkulichenko closed pull request #1: test module
vkulichenko closed pull request #1: URL: https://github.com/apache/ignite-3/pull/1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [ignite-3] vkulichenko commented on pull request #1: test module
vkulichenko commented on pull request #1: URL: https://github.com/apache/ignite-3/pull/1#issuecomment-732547333 Test PR, will close. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [VOTE] Define Apache Ignite as a Distributed Database
+1 On Mon, Nov 23, 2020 at 4:55 PM Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > +1 > > On Mon, Nov 23, 2020 at 2:44 PM Denis Magda wrote: > > > Igniters, > > > > With this vote, I'd like to formally wrap up our discussion on defining > > Ignite as a "distributed database" instead of an "in-memory computing" > > platform. See the following discussion for the rationale and context: > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Renaming-Ignite-s-product-category-td49246.html > > > > If the vote passes, the website will define Ignite as "a distributed > > database for in-memory speed at petabyte scale" to underscore our > in-memory > > origins (and the primary reason why the project is selected) but not > > diminishing our persistence capabilities. All prominent use cases such as > > caching, high-performance computing, etc. will remain visible on the > > website. There is nothing wrong if a distributed database is used as a > > cache or high-performance compute cluster; it's up to an application > > developer to decide. > > > > Overall, please cast your vote for defining Ignite as a "distributed > > database": > > +1 - support the change > > -1 - disagree with the change, explain why > > 0 - neutral > > > > This is a majority vote that is open for the next 7 days and to be closed > > on Monday, Nov 30th: > > https://www.timeanddate.com/countdown/to?year=2020=11=30 > > > > - > > Denis > > >
Re: [VOTE][EXTENSION] Release Apache Ignite Streaming extensions 1.0.0 RC1
+1 Denis, Alex I have raised PR for the docs for Maven artifactIds and versions on the documentation pages of the integrations. PR : https://github.com/apache/ignite/pull/8488 Jira : https://issues.apache.org/jira/browse/IGNITE-12951 Please review and share if any further changes are required. Regards, Saikat On Mon, Nov 23, 2020 at 5:50 PM Denis Magda wrote: > +1 > > Alex, could you please update Maven artifactIds and versions on the > documentation pages of the integrations? Commit to the master and > cherry-pick to ignite-2.9-docs, and I'll publish the changes to the > website. > > - > Denis > > > On Mon, Nov 23, 2020 at 10:52 AM Alexey Goncharuk > wrote: > > > Dear Ignite Community, > > > > I have uploaded a release candidate of the following extension modules: > > ignite-camel-ext > > ignite-flink-ext > > ignite-flume-ext > > ignite-jms11-ext > > ignite-kafka-ext > > ignite-mqtt-ext > > ignite-pub-sub-ext > > ignite-rocketmq-ext > > ignite-storm-ext > > ignite-twitter-ext > > ignite-zeromq-ext > > > > The following staging can be used for testing: > > https://repository.apache.org/content/repositories/orgapacheignite-1489 > > > > Tags were created: > > ignite-camel-ext-1.0.0-rc1 > > ignite-flink-ext-1.0.0-rc1 > > ignite-flume-ext-1.0.0-rc1 > > ignite-jms11-ext-1.0.0-rc1 > > ignite-kafka-ext-1.0.0-rc1 > > ignite-mqtt-ext-1.0.0-rc1 > > ignite-pub-sub-ext-1.0.0-rc1 > > ignite-rocketmq-ext-1.0.0-rc1 > > ignite-storm-ext-1.0.0-rc1 > > ignite-twitter-ext-1.0.0-rc1 > > ignite-zeromq-ext-1.0.0-rc1 > > > > > https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=commit;h=3f45b3e32d617b652e08af614a90aad6b9ff945d > > > > RELEASE NOTES: > > > > > https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=blob;f=RELEASE_NOTES.txt;h=8451780d53af3eea926fbd3235cc535d76888fa9;hb=refs/heads/streaming-all-ext-1.0.0 > > > > Release 1.0.0 contains the initial state of the modules migrated from the > > Apache Ignite main repository according to the IEP (includes the list of > > resolved issues): > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization > > > > The vote is formal, see voting guidelines > > https://www.apache.org/foundation/voting.html > > > > +1 - to accept all the Apache Ignite streaming extensions 1.0.0-rc1 > listed > > in the thread > > 0 - don't care either way > > -1 - DO NOT accept either of the Apache Ignite streaming extensions > > 1.0.0-rc1 (explain why) > > > > Given the large list of extensions, the vote will hold for 7 days and > will > > end on Monday, November 30th 19:00 UTC: > > > > > https://www.timeanddate.com/countdown/generic?iso=20201130T22=166=cursive=1 > > >
[GitHub] [ignite-3] vkulichenko opened a new pull request #1: test module
vkulichenko opened a new pull request #1: URL: https://github.com/apache/ignite-3/pull/1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: IEP-54: Schema-first approach for 3.0
Andrey, thanks for the update,
Does any of the serializers take into consideration the
native-image-generation feature of GraalVM?
https://www.graalvm.org/reference-manual/native-image/
With the current binary marshaller, we can't even generate a native image
for the code using our thin client APIs.
-
Denis
On Mon, Nov 23, 2020 at 4:39 AM Andrey Mashenkov
wrote:
> Hi Igniters,
>
> I'd like to continue discussion of IEP-54 (Schema-first approach).
>
> Hope everyone who is interested had a chance to get familiar with the
> proposal [1].
> Please, do not hesitate to ask questions and share your ideas.
>
> I've prepared a prototype of serializer [2] for the data layout described
> in the proposal.
> In prototy, I compared 2 approaches to (de)serialize objects, the first one
> uses java reflection/unsafe API and similar to one we already use in Ignite
> and the second one generates serializer for particular user class and uses
> Janino library for compilation.
> Second one shows better results in benchmarks.
> I think we can go with it as default serializer and have reflection-based
> implementation as a fallback if someone will have issues with the first
> one.
> WDYT?
>
> There are a number of tasks under the umbrella ticket [3] waiting for the
> assignee.
>
> BTW, I'm going to create more tickets for schema manager modes
> implementation, but would like to clarify some details.
>
> I thought schemaManager on each node should held:
> 1. Local mapping of "schema version" <--> validated local key/value
> classes pair.
> 2. Cluster-wide schema changes history.
> On the client side. Before any key-value API operation we should validate a
> schema for a given key-value pair.
> If there is no local-mapping exists for a given key-value pair or if a
> cluster wide schema has a more recent version then the key-value pair
> should be validated against the latest version and local mapping should be
> updated/actualized.
> If an object doesn't fit to the latest schema then it depends on schema
> mode: either fail the operation ('strict' mode) or a new mapping should be
> created and a new schema version should be propagated to the cluster.
>
> On the server side we usually have no key-value classes and we operate with
> tuples.
> As schema change history is available and a tuple has schema version, then
> it is possible to upgrade any received tuple to the last version without
> desialization.
> Thus we could allow nodes to send key-value pairs of previous versions (if
> they didn't receive a schema update yet) without reverting schema changes
> made by a node with newer classes.
>
> Alex, Val, Ivan did you mean the same?
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
> [2] https://github.com/apache/ignite/tree/ignite-13618/modules/commons
> [3] https://issues.apache.org/jira/browse/IGNITE-13616
>
> On Thu, Sep 17, 2020 at 9:21 AM Ivan Pavlukhin
> wrote:
>
> > Folks,
> >
> > Please do not ignore history. We had a thread [1] with many bright
> > ideas. We can resume it.
> >
> > [1]
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Applicability-of-term-cache-to-Apache-Ignite-td36541.html
> >
> > 2020-09-10 0:08 GMT+03:00, Denis Magda :
> > > Val, makes sense, thanks for explaining.
> > >
> > > Agree that we need to have a separate discussion thread for the "table"
> > and
> > > "cache" terms substitution. I'll appreciate it if you start the thread
> > > sharing pointers to any relevant IEPs and reasoning behind the
> suggested
> > > change.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Tue, Sep 8, 2020 at 6:01 PM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > >> Hi Denis,
> > >>
> > >> I guess the wording in the IEP is a little bit confusing. All it means
> > is
> > >> that you should not create nested POJOs, but rather inline fields
> into a
> > >> single POJO that is mapped to a particular schema. In other words,
> > nested
> > >> POJOs are not supported.
> > >>
> > >> Alex, is this correct? Please let me know if I'm missing something.
> > >>
> > >> As for the "cache" term, I agree that it is outdated, but I'm not sure
> > >> what we can replace it with. "Table" is tightly associated with SQL,
> but
> > >> SQL is optional in our case. Do you want to create a separate
> discussion
> > >> about this?
> > >>
> > >> -Val
> > >>
> > >> On Tue, Sep 8, 2020 at 4:37 PM Denis Magda wrote:
> > >>
> > >>> Val,
> > >>>
> > >>> I've checked the IEP again and have a few questions.
> > >>>
> > >>> Arbitrary nested objects and collections are not allowed as column
> > >>> values.
> > >>> > Nested POJOs should either be inlined into schema, or stored as
> BLOBs
> > >>>
> > >>>
> > >>> Could you provide a DDL code snippet showing how the inlining of
> POJOs
> > >>> is
> > >>> supposed to work?
> > >>>
> > >>> Also, we keep using the terms "cache" and "table" throughout the IEP.
> > Is
> > >>> it
> > >>> the right time to discuss an
Re: [VOTE][EXTENSION] Release Apache Ignite Streaming extensions 1.0.0 RC1
+1 Alex, could you please update Maven artifactIds and versions on the documentation pages of the integrations? Commit to the master and cherry-pick to ignite-2.9-docs, and I'll publish the changes to the website. - Denis On Mon, Nov 23, 2020 at 10:52 AM Alexey Goncharuk wrote: > Dear Ignite Community, > > I have uploaded a release candidate of the following extension modules: > ignite-camel-ext > ignite-flink-ext > ignite-flume-ext > ignite-jms11-ext > ignite-kafka-ext > ignite-mqtt-ext > ignite-pub-sub-ext > ignite-rocketmq-ext > ignite-storm-ext > ignite-twitter-ext > ignite-zeromq-ext > > The following staging can be used for testing: > https://repository.apache.org/content/repositories/orgapacheignite-1489 > > Tags were created: > ignite-camel-ext-1.0.0-rc1 > ignite-flink-ext-1.0.0-rc1 > ignite-flume-ext-1.0.0-rc1 > ignite-jms11-ext-1.0.0-rc1 > ignite-kafka-ext-1.0.0-rc1 > ignite-mqtt-ext-1.0.0-rc1 > ignite-pub-sub-ext-1.0.0-rc1 > ignite-rocketmq-ext-1.0.0-rc1 > ignite-storm-ext-1.0.0-rc1 > ignite-twitter-ext-1.0.0-rc1 > ignite-zeromq-ext-1.0.0-rc1 > > https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=commit;h=3f45b3e32d617b652e08af614a90aad6b9ff945d > > RELEASE NOTES: > > https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=blob;f=RELEASE_NOTES.txt;h=8451780d53af3eea926fbd3235cc535d76888fa9;hb=refs/heads/streaming-all-ext-1.0.0 > > Release 1.0.0 contains the initial state of the modules migrated from the > Apache Ignite main repository according to the IEP (includes the list of > resolved issues): > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization > > The vote is formal, see voting guidelines > https://www.apache.org/foundation/voting.html > > +1 - to accept all the Apache Ignite streaming extensions 1.0.0-rc1 listed > in the thread > 0 - don't care either way > -1 - DO NOT accept either of the Apache Ignite streaming extensions > 1.0.0-rc1 (explain why) > > Given the large list of extensions, the vote will hold for 7 days and will > end on Monday, November 30th 19:00 UTC: > > https://www.timeanddate.com/countdown/generic?iso=20201130T22=166=cursive=1 >
Re: [VOTE] Define Apache Ignite as a Distributed Database
+1 On Mon, Nov 23, 2020 at 2:44 PM Denis Magda wrote: > Igniters, > > With this vote, I'd like to formally wrap up our discussion on defining > Ignite as a "distributed database" instead of an "in-memory computing" > platform. See the following discussion for the rationale and context: > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Renaming-Ignite-s-product-category-td49246.html > > If the vote passes, the website will define Ignite as "a distributed > database for in-memory speed at petabyte scale" to underscore our in-memory > origins (and the primary reason why the project is selected) but not > diminishing our persistence capabilities. All prominent use cases such as > caching, high-performance computing, etc. will remain visible on the > website. There is nothing wrong if a distributed database is used as a > cache or high-performance compute cluster; it's up to an application > developer to decide. > > Overall, please cast your vote for defining Ignite as a "distributed > database": > +1 - support the change > -1 - disagree with the change, explain why > 0 - neutral > > This is a majority vote that is open for the next 7 days and to be closed > on Monday, Nov 30th: > https://www.timeanddate.com/countdown/to?year=2020=11=30 > > - > Denis >
[VOTE] Define Apache Ignite as a Distributed Database
Igniters, With this vote, I'd like to formally wrap up our discussion on defining Ignite as a "distributed database" instead of an "in-memory computing" platform. See the following discussion for the rationale and context: http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Renaming-Ignite-s-product-category-td49246.html If the vote passes, the website will define Ignite as "a distributed database for in-memory speed at petabyte scale" to underscore our in-memory origins (and the primary reason why the project is selected) but not diminishing our persistence capabilities. All prominent use cases such as caching, high-performance computing, etc. will remain visible on the website. There is nothing wrong if a distributed database is used as a cache or high-performance compute cluster; it's up to an application developer to decide. Overall, please cast your vote for defining Ignite as a "distributed database": +1 - support the change -1 - disagree with the change, explain why 0 - neutral This is a majority vote that is open for the next 7 days and to be closed on Monday, Nov 30th: https://www.timeanddate.com/countdown/to?year=2020=11=30 - Denis
[VOTE][EXTENSION] Release Apache Ignite Streaming extensions 1.0.0 RC1
Dear Ignite Community, I have uploaded a release candidate of the following extension modules: ignite-camel-ext ignite-flink-ext ignite-flume-ext ignite-jms11-ext ignite-kafka-ext ignite-mqtt-ext ignite-pub-sub-ext ignite-rocketmq-ext ignite-storm-ext ignite-twitter-ext ignite-zeromq-ext The following staging can be used for testing: https://repository.apache.org/content/repositories/orgapacheignite-1489 Tags were created: ignite-camel-ext-1.0.0-rc1 ignite-flink-ext-1.0.0-rc1 ignite-flume-ext-1.0.0-rc1 ignite-jms11-ext-1.0.0-rc1 ignite-kafka-ext-1.0.0-rc1 ignite-mqtt-ext-1.0.0-rc1 ignite-pub-sub-ext-1.0.0-rc1 ignite-rocketmq-ext-1.0.0-rc1 ignite-storm-ext-1.0.0-rc1 ignite-twitter-ext-1.0.0-rc1 ignite-zeromq-ext-1.0.0-rc1 https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=commit;h=3f45b3e32d617b652e08af614a90aad6b9ff945d RELEASE NOTES: https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=blob;f=RELEASE_NOTES.txt;h=8451780d53af3eea926fbd3235cc535d76888fa9;hb=refs/heads/streaming-all-ext-1.0.0 Release 1.0.0 contains the initial state of the modules migrated from the Apache Ignite main repository according to the IEP (includes the list of resolved issues): https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization The vote is formal, see voting guidelines https://www.apache.org/foundation/voting.html +1 - to accept all the Apache Ignite streaming extensions 1.0.0-rc1 listed in the thread 0 - don't care either way -1 - DO NOT accept either of the Apache Ignite streaming extensions 1.0.0-rc1 (explain why) Given the large list of extensions, the vote will hold for 7 days and will end on Monday, November 30th 19:00 UTC: https://www.timeanddate.com/countdown/generic?iso=20201130T22=166=cursive=1
Re: Hello [ New to apache ignite]
Hello Amit, Welcome to the community! Glad to see you are ready to contribute to Ignite. Please check the "Code Contributions" and "Contributing to Ignite Core" sections on this page: https://ignite.apache.org/community/contribute.html We put and grouped together some tasks that you might be interested in. Also, if you haven't used Ignite before, try to launch and build the first applications becoming familiar with the capabilities of the project: https://ignite.apache.org/docs/latest/quick-start/java Send us any questions you might have. - Denis On Sun, Nov 22, 2020 at 4:07 AM Amit Chavan wrote: > Hello everyone, > > My name is Amit Chavan. I found the community after I watched [Live coding > distributed system in java] on youtube. A little bit about myself. I have > over 10 years of development experience in ad-tech industry. Recently I > joined the AWS S3 team in the user security space based on VA. I am > interested in writing protocol servers, distributed systems and database. > Outside of work I contribute to some open-source projects such as Armeria > and herddb. I can start anywhere on the project where help is needed. I can > pick up some newbie issues to familiarize myself. My jira-id is > achav...@gmail.com. I look forward to working with everyone. > > Thanks, > Amit >
Re: IEP-54: Schema-first approach for 3.0
Hi Igniters,
I'd like to continue discussion of IEP-54 (Schema-first approach).
Hope everyone who is interested had a chance to get familiar with the
proposal [1].
Please, do not hesitate to ask questions and share your ideas.
I've prepared a prototype of serializer [2] for the data layout described
in the proposal.
In prototy, I compared 2 approaches to (de)serialize objects, the first one
uses java reflection/unsafe API and similar to one we already use in Ignite
and the second one generates serializer for particular user class and uses
Janino library for compilation.
Second one shows better results in benchmarks.
I think we can go with it as default serializer and have reflection-based
implementation as a fallback if someone will have issues with the first one.
WDYT?
There are a number of tasks under the umbrella ticket [3] waiting for the
assignee.
BTW, I'm going to create more tickets for schema manager modes
implementation, but would like to clarify some details.
I thought schemaManager on each node should held:
1. Local mapping of "schema version" <--> validated local key/value
classes pair.
2. Cluster-wide schema changes history.
On the client side. Before any key-value API operation we should validate a
schema for a given key-value pair.
If there is no local-mapping exists for a given key-value pair or if a
cluster wide schema has a more recent version then the key-value pair
should be validated against the latest version and local mapping should be
updated/actualized.
If an object doesn't fit to the latest schema then it depends on schema
mode: either fail the operation ('strict' mode) or a new mapping should be
created and a new schema version should be propagated to the cluster.
On the server side we usually have no key-value classes and we operate with
tuples.
As schema change history is available and a tuple has schema version, then
it is possible to upgrade any received tuple to the last version without
desialization.
Thus we could allow nodes to send key-value pairs of previous versions (if
they didn't receive a schema update yet) without reverting schema changes
made by a node with newer classes.
Alex, Val, Ivan did you mean the same?
[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
[2] https://github.com/apache/ignite/tree/ignite-13618/modules/commons
[3] https://issues.apache.org/jira/browse/IGNITE-13616
On Thu, Sep 17, 2020 at 9:21 AM Ivan Pavlukhin wrote:
> Folks,
>
> Please do not ignore history. We had a thread [1] with many bright
> ideas. We can resume it.
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/Applicability-of-term-cache-to-Apache-Ignite-td36541.html
>
> 2020-09-10 0:08 GMT+03:00, Denis Magda :
> > Val, makes sense, thanks for explaining.
> >
> > Agree that we need to have a separate discussion thread for the "table"
> and
> > "cache" terms substitution. I'll appreciate it if you start the thread
> > sharing pointers to any relevant IEPs and reasoning behind the suggested
> > change.
> >
> > -
> > Denis
> >
> >
> > On Tue, Sep 8, 2020 at 6:01 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> >> Hi Denis,
> >>
> >> I guess the wording in the IEP is a little bit confusing. All it means
> is
> >> that you should not create nested POJOs, but rather inline fields into a
> >> single POJO that is mapped to a particular schema. In other words,
> nested
> >> POJOs are not supported.
> >>
> >> Alex, is this correct? Please let me know if I'm missing something.
> >>
> >> As for the "cache" term, I agree that it is outdated, but I'm not sure
> >> what we can replace it with. "Table" is tightly associated with SQL, but
> >> SQL is optional in our case. Do you want to create a separate discussion
> >> about this?
> >>
> >> -Val
> >>
> >> On Tue, Sep 8, 2020 at 4:37 PM Denis Magda wrote:
> >>
> >>> Val,
> >>>
> >>> I've checked the IEP again and have a few questions.
> >>>
> >>> Arbitrary nested objects and collections are not allowed as column
> >>> values.
> >>> > Nested POJOs should either be inlined into schema, or stored as BLOBs
> >>>
> >>>
> >>> Could you provide a DDL code snippet showing how the inlining of POJOs
> >>> is
> >>> supposed to work?
> >>>
> >>> Also, we keep using the terms "cache" and "table" throughout the IEP.
> Is
> >>> it
> >>> the right time to discuss an alternate name that would replace those
> >>> too?
> >>> Personally, the "table" should stay and the "cache" should go
> >>> considering
> >>> that SQL is one of the primary APIs in Ignite and that DDL is supported
> >>> out-of-the-box.
> >>>
> >>>
> >>> -
> >>> Denis
> >>>
> >>>
> >>> On Mon, Sep 7, 2020 at 12:26 PM Valentin Kulichenko <
> >>> valentin.kuliche...@gmail.com> wrote:
> >>>
> >>> > Ivan,
> >>> >
> >>> > I see your point. I agree that with the automatic updates we step
> into
> >>> the
> >>> > schema-last territory.
> >>> >
> >>> > Actually, if we support automatic evolution, we can as well support
>
[MTCGA]: new failures in builds [5752528] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. *New Critical Failure in master Thin client: Node.js https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_ThinClientNodeJs?branch=%3Cdefault%3E No changes in the build - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 15:22:39 23-11-2020
[jira] [Created] (IGNITE-13745) Add release notes for streaming extensions release 1.0.0
Alexey Goncharuk created IGNITE-13745: - Summary: Add release notes for streaming extensions release 1.0.0 Key: IGNITE-13745 URL: https://issues.apache.org/jira/browse/IGNITE-13745 Project: Ignite Issue Type: Sub-task Reporter: Alexey Goncharuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[MTCGA]: new failures in builds [5745961] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. *New Critical Failure in master Thin client: Node.js https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_ThinClientNodeJs?branch=%3Cdefault%3E No changes in the build - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 14:52:40 23-11-2020
[MTCGA]: new failures in builds [5745959] needs to be handled
Hi Igniters, I've detected some new issue on TeamCity to be handled. You are more than welcomed to help. *New Critical Failure in master Thin client: Node.js https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_ThinClientNodeJs?branch=%3Cdefault%3E No changes in the build - Here's a reminder of what contributors were agreed to do https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute - Should you have any questions please contact dev@ignite.apache.org Best Regards, Apache Ignite TeamCity Bot https://github.com/apache/ignite-teamcity-bot Notification generated at 14:37:40 23-11-2020
Re: IEP-61 Technical discussion
Thanks, Ivan, Another protocol for group membership worth checking out is RAPID [1] (a recent one). Not sure though if there are any available implementations for it already. [1] https://www.usenix.org/system/files/conference/atc18/atc18-suresh.pdf пн, 23 нояб. 2020 г. в 10:46, Ivan Daschinsky : > Also, here is some interesting reading about gossip, SWIM etc. > > 1 -- > http://www.cs.cornell.edu/Info/Projects/Spinglass/public_pdfs/SWIM.pdf > 2 -- > http://www.antonkharenko.com/2015/09/swim-distributed-group-membership.html > 3 -- https://github.com/hashicorp/memberlist (Foundation library of > hashicorp serf) > 4 -- https://github.com/scalecube/scalecube-cluster -- (Java > implementation > of SWIM) > > чт, 19 нояб. 2020 г. в 16:35, Ivan Daschinsky : > > > >> Friday, Nov 27th work for you? If ok, let's have an open call then. > > Yes, great > > >> As for the protocol port - we will not be dealing with the > > concurrency... > > >>Judging by the Rust port, it seems fairly straightforward. > > Yes, they chose split transport and logic. But original Go package from > > etcd (see raft/node.go) contains some heartbeats mechanism etc. > > I agree with you, this seems not to be a huge deal to port. > > > > чт, 19 нояб. 2020 г. в 16:13, Alexey Goncharuk < > alexey.goncha...@gmail.com > > >: > > > >> Ivan, > >> > >> Agree, let's have a call to discuss the IEP. I have some more thoughts > >> regarding how the replication infrastructure works with > >> atomic/transactional caches, will put this info to the IEP. Does next > >> Friday, Nov 27th work for you? If ok, let's have an open call then. > >> > >> As for the protocol port - we will not be dealing with the concurrency > >> model if we choose this way, this is what I like about their code > >> structure. Essentially, the raft module is a single-threaded automata > >> which > >> has a callback to process a message, process a tick (timeout) and > produces > >> messages that should be sent and log entries that should be persisted. > >> Judging by the Rust port, it seems fairly straightforward. Will be happy > >> to > >> discuss this and other alternatives on the call as well. > >> > >> чт, 19 нояб. 2020 г. в 14:41, Ivan Daschinsky : > >> > >> > > Any existing library that can be used to avoid re-implementing the > >> > protocol ourselves? Perhaps, porting the existing implementation to > Java > >> > Personally, I like this idea. Go libraries (either raft module of etcd > >> or > >> > serf by Hashicorp) are famous for clean code, good design, stability, > >> not > >> > enormous size. > >> > But, on other side, Go has different model for concurrency and porting > >> > probably will not be so straightforward. > >> > > >> > > >> > > >> > чт, 19 нояб. 2020 г. в 13:48, Ivan Daschinsky : > >> > > >> > > I'd suggest to discuss this IEP and technical details in open ZOOM > >> > > meeting. > >> > > > >> > > чт, 19 нояб. 2020 г. в 13:47, Ivan Daschinsky >: > >> > > > >> > >> > >> > >> > >> > >> -- Forwarded message - > >> > >> От: Ivan Daschinsky > >> > >> Date: чт, 19 нояб. 2020 г. в 13:02 > >> > >> Subject: Re: IEP-61 Technical discussion > >> > >> To: Alexey Goncharuk > >> > >> > >> > >> > >> > >> Alexey, let's arise another question. Specifically, how nodes > >> initially > >> > >> find each other (discovery) and how they detect failures. > >> > >> > >> > >> I suppose, that gossip protocol is an ideal candidate. For example, > >> > >> consul [1] uses this approach, using serf [2] library to discover > >> > members > >> > >> of cluster. > >> > >> Then consul forms raft ensemble (server nodes) and client use raft > >> > >> ensemble only as lock service. > >> > >> > >> > >> PacificA suggests internal heartbeats mechanism for failure > >> detection of > >> > >> replicated group, but it says nothing about initial discovery of > >> nodes. > >> > >> > >> > >> WDYT? > >> > >> > >> > >> [1] -- https://www.consul.io/docs/architecture/gossip > >> > >> [2] -- https://www.serf.io/ > >> > >> > >> > >> чт, 19 нояб. 2020 г. в 12:46, Alexey Goncharuk < > >> > >> alexey.goncha...@gmail.com>: > >> > >> > >> > >>> Following up the Ignite 3.0 scope/development approach threads, > >> this is > >> > >>> a separate thread to discuss technical aspects of the IEP. > >> > >>> > >> > >>> Let's reiterate one more time on the questions raised by Ivan and > >> also > >> > >>> see if there are any other thoughts on the IEP: > >> > >>> > >> > >>>- *Whether to deploy metastorage on a separate subset of the > >> nodes > >> > >>>or allow Ignite to choose these nodes automatically.* I think > it > >> is > >> > >>>feasible to maintain both modes: by default, Ignite will choose > >> > >>>metastorage nodes automatically which essentially will provide > >> the > >> > same > >> > >>>seamless user experience as TCP discovery SPI - no separate > >> roles, > >> > >>>simplistic deployment. For deployments where people want to > have > >> > more > >> > >>>fine-grained
Re: Hello [ New to apache ignite]
Hello! It seems that you were already assigned a Contributor role. Please try assigning tickets to yourself. Please read https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines and familiarize yourself with https://mtcga.gridgain.com/ bot. Regards, -- Ilya Kasnacheev вс, 22 нояб. 2020 г. в 15:08, Amit Chavan : > Hello everyone, > > My name is Amit Chavan. I found the community after I watched [Live coding > distributed system in java] on youtube. A little bit about myself. I have > over 10 years of development experience in ad-tech industry. Recently I > joined the AWS S3 team in the user security space based on VA. I am > interested in writing protocol servers, distributed systems and database. > Outside of work I contribute to some open-source projects such as Armeria > and herddb. I can start anywhere on the project where help is needed. I can > pick up some newbie issues to familiarize myself. My jira-id is > achav...@gmail.com. I look forward to working with everyone. > > Thanks, > Amit >
[jira] [Created] (IGNITE-13744) Use Table spool for IgniteNestedLoopJoin
Taras Ledkov created IGNITE-13744:
-
Summary: Use Table spool for IgniteNestedLoopJoin
Key: IGNITE-13744
URL: https://issues.apache.org/jira/browse/IGNITE-13744
Project: Ignite
Issue Type: Bug
Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
Now {{NestedLoopJoinNode}} uses internal buffer to save all rows of the right
input.
We have to do refactoring {{IgniteNestedLoopJoin}} to use rewind of the right
input and use TableSpool for not rewindable inputs.
This refactoring separates implementation the join logic from materialization
of the right input if it is needed. In the future we can use disk offload for
TableSpool etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13743) Defragmentation JMX API for schedule/cancel/status
Ivan Bessonov created IGNITE-13743: -- Summary: Defragmentation JMX API for schedule/cancel/status Key: IGNITE-13743 URL: https://issues.apache.org/jira/browse/IGNITE-13743 Project: Ignite Issue Type: Sub-task Reporter: Ivan Bessonov Assignee: Semyon Danilov -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re[2]: [DISCUSS] Page replacement improvement
Nikolay, i hope such case ignite users already observed) I suggest to: put data bigger then available, full scan, get data only for available inmem data in loop, check PageReplacement metric, how match iterations will bring to zero this metric. >Hello, Alex. > >> Perhaps we can implement a special benchmark for this case with manually >> triggered "batch page replacement" using yardstick (but I'm not sure if >> ducktape can help here). > >I think we should carefully describe the issues with the current approach and >then we can choose right tool to benchmark improvements. >You said: > >> we use Random-LRU algorithm … it has many disadvantages and affects >> performance very much when replacement is started > >Can you list disadvantages of the Random-LRU? > >AFAIU the first benchmark should be: > >1. Start cluster with persistence and put data bigger then available RAM to it. >2. Measure performance of the queries that selects one entry from the cache. >3. Make some queries that will iterate throw all data - `SELECT SUM(x) FROM t` >or something similar. >4. Repeat step 2. in this time performance of random queries should be lower >due to the page replacement. > >Is this scenario correct? > >> 23 нояб. 2020 г., в 09:12, Alex Plehanov < plehanov.a...@gmail.com > >> написал(а): >> >> Nikolay, Zhenya, >> >> Benchmark from IGNITE-13034 is fully synthetic, it makes random puts >> uniformly. It can be used to compare different page replacement algorithms >> (random-LRU vs segmented-LRU), but it is totally inapplicable to benchmark >> batch page replacement. >> Perhaps we can implement a special benchmark for this case with manually >> triggered "batch page replacement" using yardstick (but I'm not sure >> if ducktape can help here). >> >>> I understand case you described, but who will pull the switch ? Human, >> artificial intelligence ? >> As I described before, we can implement both manual and automatic "batch >> page replacement" triggering. For automatic triggering, there is no >> artificial intelligence needed, just several conditions with reasonable >> thresholds. Automatic triggering also can be disabled by default. >> >> пт, 20 нояб. 2020 г. в 13:32, Zhenya Stanilovsky < arzamas...@mail.ru.invalid >>> : >> >>> >>> >>> >>> >>> >>> Zhenya, > Alexey, we already have changes that partially fixes this issue [1] IGNITE-13086 it's a minor improvement. We still have major problems with our page replacement algorithm (slow page selection and non-optimal page-fault rate). I think changing from random 5 pages to 7 will make things even worse (it's better for page-fault rate, but page selection >>> will be slower). >>> All this words above need to be proven, i hope. + 1 with Nikolay, we need >>> correct reproduces or some graphs from 2.9 ver. >>> > This approach still not applicable for real life Why do you think batch replacement is not applicable for real-life? It can be applied for workloads, where some big amount of data periodically used, but not very often. For example, when OLAP request over historical data raised pages to page-memory, and after such request this data is not >>> needed for a long time. Or when OLTP transactions mostly add new data and process recent data but rarely touch historical data. In these cases with the current approach, we will enter "page replacement mode" after some period of time and never leave it. With batch page replacement there is a chance to prevent random-LRU page replacement or postpone it. >>> I understand case you described, but who will pull the switch ? Human, >>> artificial intelligence ? >>> You approach assume some triggering from inner, i don`t like this. >>> > But request once more, do you really observe such problems with 2.9 ver >>> ? Any graphs maybe ? I don't have production usage feedback after IGNITE-13086, but I doubt something changed significantly. >>> >>> Lets wait ?:) In any case (Nikolay, Alex) IGNITE-13086 includes yardstik >>> bench for PR proven, we can use it once more. >>> >>> Thanks ! чт, 19 нояб. 2020 г. в 09:18, Zhenya Stanilovsky < >>> arzamas...@mail.ru.invalid > : > > Alexey, we already have changes that partially fixes this issue [1] > Easy way: > Looks like we already have converge in page replacement. > If we change 5 times touch iterator from random lru algo into, for > example — 7 we will obtain fast improvement from scratch. > > » Batch page replacement > This approach still not applicable for real life if you wan`t to observe > ugly people for threshold (i.e. 12 h) interval. And, of course, you > understand that dramatically reduce of such interval gives nothing? > > » Change the page replacement algorithm. > That`s way i vote for ) But request once more, do you really observe >>> such > problems with 2.9 ver ? Any
