Re: Forbid mixed cache groups with both atomic and transactional caches
Folks, Thanks for your feedback. I've created a JIRA issue on this change: https://issues.apache.org/jira/browse/IGNITE-12622 On Tue, Feb 4, 2020 at 10:43 PM Denis Magda wrote: > +1 from my end. It doesn't sound like a big deal if Ignite users need to > define separate groups for atomic and transactional caches. > > - > Denis > > > On Tue, Feb 4, 2020 at 3:28 AM Ivan Rakov wrote: > > > Igniters, > > > > Apparently it's possible in Ignite to configure a cache group with both > > ATOMIC and TRANSACTIONAL caches. > > Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests. > > In my opinion, it would be better to remove such possibility from the > > product. There are several reasons: > > > > 1) The original idea of grouping caches was optimizing storage overhead > and > > PME time by joining data of similar caches into the same partitions. > ATOMIC > > and TRANSACTIONAL caches provide different guarantees and are designed > for > > different use cases, thus they can hardly be called "similar". > > > > 2) Diving deeper: synchronization protocols and possible reasons for > > primary-backup divergences are conceptually different for ATOMIC and > > TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery > protocol > > allows to recover consistency if any participating node will fail, but > for > > ATOMIC caches there's possible scenario with failure of primary node > where > > neither of backups will contain the most recent state of the data. > Example: > > one backup have received updates 1, 3, 5 while another have received 2, 4 > > (which is possible due to message reordering), and even tracking counters > > [1] won't restore the consistency. The problem is that we can't > distinguish > > what kind of conflict we have faced in case update counters have diverged > > in a mixed group. > > > > 3) Mixed groups are poorly tested. I can't find any tests except a couple > > of smoke tests in IgniteCacheGroupsTest. We can't be sure that different > > synchronization protocols will work correctly for such configurations, > > especially under load and with a variety of dependent configuration > > parameters. > > > > 4) I have never heard of any feedback on mixed groups. I have asked > > different people on this and no one recalled any attempts to configure > such > > groups. I believe that in fact no one has ever tried to do it. > > > > Please let me know if you are aware of any cases where mixed groups are > > used or reasons to keep them. Otherwise I'll create a ticket to prohibit > > mixed configurations. > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-11797 > > > > -- > > Best Regards, > > Ivan Rakov > > >
[jira] [Created] (IGNITE-12622) Forbid mixed cache groups with both atomic and transactional caches
Ivan Rakov created IGNITE-12622: --- Summary: Forbid mixed cache groups with both atomic and transactional caches Key: IGNITE-12622 URL: https://issues.apache.org/jira/browse/IGNITE-12622 Project: Ignite Issue Type: Improvement Components: cache Reporter: Ivan Rakov Fix For: 2.9 Apparently it's possible in Ignite to configure a cache group with both ATOMIC and TRANSACTIONAL caches. IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests. As per discussed on dev list (http://apache-ignite-developers.2346864.n4.nabble.com/Forbid-mixed-cache-groups-with-both-atomic-and-transactional-caches-td45586.html), the community has concluded that such configurations should be prohibited. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Internal classes are exposed in public API
Let's run a vote if that's the only option to come to a consensus. It will be best if either Alex Goncharuk, Andrey Gura or Nikolay Izhikov creates a discussion thread stating the problem and possible choices. Folks, who would like to take over? Generally, the vote should help us to decide how to deal with similar situations in the future. We need to agree on how and in what circumstances we deprecate existing APIs and engrave this on our wiki. As for this discussion related to the new metrics framework, it should be used just as a reference. - Denis On Tue, Feb 4, 2020 at 2:38 AM Maxim Muzafarov wrote: > Folks, > > Let's start a vote? > > On Mon, 3 Feb 2020 at 15:05, Andrey Gura wrote: > > > > Just post here article from Oracle documentation "How and When To > > Deprecate APIs" [1]. > > > > [1] > https://docs.oracle.com/javase/8/docs/technotes/guides/javadoc/deprecation/deprecation.html > > > > On Fri, Jan 31, 2020 at 10:44 AM Pavel Tupitsyn > wrote: > > > > > > Agree with Andrey, let's remove deprecation for now and unblock the > release. > > > > > > On Thu, Jan 30, 2020 at 10:23 PM Andrey Gura wrote: > > > > > > > I'll just repeat one thought with some changes. > > > > > > > > There are at least two groups of people in this discussion. One group > > > > sure that new API is complete and production ready while other group > > > > disagree with it. Obviously we can't reach consensus about it right > > > > now. But we can do it in the future. > > > > At present we just can't deprecate existing API and mark new API as > > > > experimental at the same time. So we must remove deprecation until > the > > > > consensus is reached. > > > > > > > > Also there is the third group of people. This people aren't related > > > > with the API, they may be don't need the API and they wait for bug > > > > fixes and other features. It is very easy to satisfy third group: > just > > > > cut off what caused the release blocking. But it is much easier to > > > > remove @deprecated annotations. > > > > > > > > On Thu, Jan 30, 2020 at 8:54 PM Nikolay Izhikov > > > > > wrote: > > > > > > > > > > Alexey. > > > > > > > > > > I answered to your examples and issues you provide. > > > > > But, it seems the discussion of the API and the Java code itself > is not > > > > the goal of this thread anymore. > > > > > > > > > > > Should we provide a way to know the number of metrics and > registries > > > > in advance? > > > > > > > > > > No. > > > > > If you think this is the real use-as let’s add `size` methods. > > > > > It will be the simple API *extension*. > > > > > > > > > > > There is no separation on public and internal metrics > > > > > > > > > > Any metric can be changed(removed) in any time. > > > > > But we will try not to do it unless we have a strong reason. > > > > > > > > > > > Will we allow users to register their own metrics? > > > > > > > > > > No. > > > > > > > > > > > It's still not clear how a user will map old interfaces and > methods to > > > > the new metric names. > > > > > > > > > > We should write this information in the deprecation message. > > > > > > > > > > > 30 янв. 2020 г., в 20:27, Alexey Goncharuk < > alexey.goncha...@gmail.com> > > > > написал(а): > > > > > > > > > > > > Nikita, > > > > > > > > > > > > Disagree here. I already gave an example in this thread of how > you > > > > need to > > > > > > peek into configuration in order to obtain an instance of > exporter SPI > > > > > > which may not necessarily be the type you need. That's why > > > > IGNITE-12553 was > > > > > > created in the first place. > > > > > > > > > >
Re: Data vanished from cluster after INACTIVE/ACTIVE switch
That's the best solution for this scenario. Should we readjust the already created ticket [1] suggesting to implement the changes of Alex Scherbakov instead? [1] https://issues.apache.org/jira/browse/IGNITE-12614 - Denis On Mon, Feb 3, 2020 at 11:54 PM Alexei Scherbakov < alexey.scherbak...@gmail.com> wrote: > Folks, > > For a long time we have a flag [1] > > It does almost what we want here. > > I suggest to make this behavior default and rework it to keep data in > memory as well (we already have special "recovery" mode for caches). > > [1] > org.apache.ignite.IgniteSystemProperties#IGNITE_REUSE_MEMORY_ON_DEACTIVATE > > > > пн, 3 февр. 2020 г. в 18:47, Alexey Goncharuk >: > > > I do not mind making this change if we explicitly and clearly define what > > 'new inactive state' means. What should happen if a partition is lost in > > inactive state? What if such node joins back the cluster after? Etc. > > > > пт, 31 янв. 2020 г. в 20:57, Denis Magda : > > > > > Back up Ivan's opinion here. Initially, the activation/deactivation was > > > created for the baseline topology designed for cases with native > > > persistence. My thinking was that the mechanism itended to prevent data > > > inconsistencies while nodes with data on the disk will be in the > process > > of > > > joining the cluster. > > > > > > Artem, could you please update the docs bringing this to the attention > of > > > the user community? > > > https://issues.apache.org/jira/browse/IGNITE-12615 > > > > > > AG, what if we don't purge data from memory at least for the caches not > > > backed by the native persistence? Is this a big deal? We can certainly > > put > > > this off by my guts feel we'll return to this question sooner or later. > > > > > > - > > > Denis > > > > > > > > > On Fri, Jan 31, 2020 at 2:17 AM Ivan Pavlukhin > > > wrote: > > > > > > > For me it looks like some coincidence effect. I understand that we > get > > > > such behavior because deactivation works the same way as for > > > > persistent caches. Was cluster activation/deactivation designed and > > > > described for in-memory caches? Current behavior sounds for me a as > > > > big risk. I expect a lot of upset users unexpectedly purged all their > > > > data. > > > > > > > > пт, 31 янв. 2020 г. в 00:00, Alexey Goncharuk < > > > alexey.goncha...@gmail.com > > > > >: > > > > > > > > > > Because originally the sole purpose of deactivation was resource > > > > > deallocation. > > > > > > > > > > чт, 30 янв. 2020 г. в 22:13, Denis Magda : > > > > > > > > > > > Such a revelation for me that data is purged from RAM if someone > > > > > > deactivates the cluster. Alex, do you remember why we decided to > > > > implement > > > > > > it this way initially? > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > On Thu, Jan 30, 2020 at 2:09 AM Alexey Goncharuk < > > > > > > alexey.goncha...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > I agree on CLI and JMX because those interfaces can be used by > a > > > > system > > > > > > > administrator and can be invoked by mistake. > > > > > > > > > > > > > > As for the Java API, personally, I find it strange to add > 'force' > > > or > > > > > > > 'confirm' flags to it because it is very unlikely that such an > > > > invocation > > > > > > > is done by mistake. Such mistakes are caught during the testing > > > > phase and > > > > > > > developers will end up hard-coding 'true' as a flag anyways. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > Ivan Pavlukhin > > > > > > > > > > > > -- > > Best regards, > Alexei Scherbakov >
Re: Forbid mixed cache groups with both atomic and transactional caches
+1 from my end. It doesn't sound like a big deal if Ignite users need to define separate groups for atomic and transactional caches. - Denis On Tue, Feb 4, 2020 at 3:28 AM Ivan Rakov wrote: > Igniters, > > Apparently it's possible in Ignite to configure a cache group with both > ATOMIC and TRANSACTIONAL caches. > Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests. > In my opinion, it would be better to remove such possibility from the > product. There are several reasons: > > 1) The original idea of grouping caches was optimizing storage overhead and > PME time by joining data of similar caches into the same partitions. ATOMIC > and TRANSACTIONAL caches provide different guarantees and are designed for > different use cases, thus they can hardly be called "similar". > > 2) Diving deeper: synchronization protocols and possible reasons for > primary-backup divergences are conceptually different for ATOMIC and > TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery protocol > allows to recover consistency if any participating node will fail, but for > ATOMIC caches there's possible scenario with failure of primary node where > neither of backups will contain the most recent state of the data. Example: > one backup have received updates 1, 3, 5 while another have received 2, 4 > (which is possible due to message reordering), and even tracking counters > [1] won't restore the consistency. The problem is that we can't distinguish > what kind of conflict we have faced in case update counters have diverged > in a mixed group. > > 3) Mixed groups are poorly tested. I can't find any tests except a couple > of smoke tests in IgniteCacheGroupsTest. We can't be sure that different > synchronization protocols will work correctly for such configurations, > especially under load and with a variety of dependent configuration > parameters. > > 4) I have never heard of any feedback on mixed groups. I have asked > different people on this and no one recalled any attempts to configure such > groups. I believe that in fact no one has ever tried to do it. > > Please let me know if you are aware of any cases where mixed groups are > used or reasons to keep them. Otherwise I'll create a ticket to prohibit > mixed configurations. > > [1]: https://issues.apache.org/jira/browse/IGNITE-11797 > > -- > Best Regards, > Ivan Rakov >
Re: Contribution
> Andrey, then I suggest firstly check if old annotations are present, and if
> not - are there any new annotations. Do you agree?
I'm afraid I don't understand. Old annotations are present and new
annotations don't exist yet :)
> And should I also replace usage of old annotations or leave it as it is?
It would be great to replace old annotations by new ones.
On Tue, Feb 4, 2020 at 6:09 PM Лев Киселев wrote:
>
> Andrey, then I suggest firstly check if old annotations are present, and if
> not - are there any new annotations. Do you agree?
> And should I also replace usage of old annotations or leave it as it is?
>
> вт, 4 февр. 2020 г. в 16:57, Andrey Gura :
>
> > Lev,
> >
> > > I'm confused about your first comment. Does it mean, that the only thing
> > I
> > can do is to create new annotation?
> >
> > Yes, it does. But in this case you can mark old annotations as
> > deprecated because new annotations are real alternative.
> >
> > > So, I can't change implementation of *getDescription / getParameterName
> > > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> > > IgniteStandardMXBean* class,
> > > and replace current annotations with new one.
> >
> > You have to support both old and new annotations due to a backward
> > compatibility.
> >
> > On Tue, Feb 4, 2020 at 3:14 PM Лев Киселев
> > wrote:
> > >
> > > Thanks for answer, Andrey.
> > > I'm confused about your first comment. Does it mean, that the only thing
> > I
> > > can do is to create new annotation?
> > > Because, as you said, I must leave unchanged all related to old
> > annotations
> > > functionality.
> > > So, I can't change implementation of *getDescription / getParameterName
> > > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> > > IgniteStandardMXBean* class,
> > > and replace current annotations with new one.
> > >
> > > вт, 4 февр. 2020 г. в 14:28, Andrey Gura :
> > >
> > > > Hi, Lev!
> > > >
> > > > I think you understand task correctly.
> > > >
> > > > A couple of comments:
> > > >
> > > > - Existing annotations and related functionality must not be deleted
> > > > because it is part of public API and we committed to have stable API
> > > > between major releases.
> > > > - I would suggest introduce @MXBeanParameter annotation instead of
> > > > @MXBeanParameterInformation. Just shorten name.
> > > >
> > > > On Thu, Jan 30, 2020 at 8:53 PM Denis Magda wrote:
> > > > >
> > > > > Nikolay, Andrey,
> > > > >
> > > > > Would you be the best committers to help out here? You are already
> > > > deeply involved in metrics development and can quickly suggest Lev how
> > to
> > > > proceed with this task.
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Thu, Jan 30, 2020 at 2:46 AM Лев Киселев <
> > lev.kiselev.1...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> Hello everyone, I have question about following task:
> > > > >> [https://issues.apache.org/jira/browse/IGNITE-10698]
> > > > >> Solution proposed in task description is seem to be logical.
> > > > >> So, I need to every replace @MXBeanParametersNames and
> > > > >> @MXBeanParametersDescriptions (everywhere, for uniformity) with
> > > > something
> > > > >> like:
> > > > >> void methodName(@MXBeanParameterInformation(name = "name",
> > description =
> > > > >> "description") firstParameter, ...) {}.
> > > > >> And, of course, need to change processing logic at
> > > > >> getParameterName/getDescription methods from IgniteStandardMXBean.
> > > > >> Do I understand correctly what needs to be done?
> > > >
> >
[jira] [Created] (IGNITE-12621) Node leave may cause NullPointerException during IO message processing if security is enabled
Vyacheslav Koptilin created IGNITE-12621:
Summary: Node leave may cause NullPointerException during IO
message processing if security is enabled
Key: IGNITE-12621
URL: https://issues.apache.org/jira/browse/IGNITE-12621
Project: Ignite
Issue Type: Bug
Reporter: Vyacheslav Koptilin
Assignee: Vyacheslav Koptilin
Fix For: 2.9
In case the node will receive IO message from a dead node *after* receiving
discovery message about node fail, {{ctx.discovery().node(uuid)}} will return
{{null}}, which in turn will cause {{NullPointerException}}.
We can fix it by peeking disco cache history for retrieving attributes of the
dead node.
See:
{code}
/** {@inheritDoc} */
@Override public OperationSecurityContext withContext(UUID nodeId) {
return withContext(
secCtxs.computeIfAbsent(nodeId,
uuid -> nodeSecurityContext(
marsh, U.resolveClassLoader(ctx.config()),
ctx.discovery().node(uuid)
)
)
);
}
{code}
{noformat}
Caused by: java.lang.NullPointerException
at
org.apache.ignite.internal.processors.security.SecurityUtils.nodeSecurityContext(SecurityUtils.java:135)
at
org.apache.ignite.internal.processors.security.IgniteSecurityProcessor.lambda$withContext$0(IgniteSecurityProcessor.java:112)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.ignite.internal.processors.security.IgniteSecurityProcessor.withContext(IgniteSecurityProcessor.java:111)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Contribution
Andrey, then I suggest firstly check if old annotations are present, and if
not - are there any new annotations. Do you agree?
And should I also replace usage of old annotations or leave it as it is?
вт, 4 февр. 2020 г. в 16:57, Andrey Gura :
> Lev,
>
> > I'm confused about your first comment. Does it mean, that the only thing
> I
> can do is to create new annotation?
>
> Yes, it does. But in this case you can mark old annotations as
> deprecated because new annotations are real alternative.
>
> > So, I can't change implementation of *getDescription / getParameterName
> > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> > IgniteStandardMXBean* class,
> > and replace current annotations with new one.
>
> You have to support both old and new annotations due to a backward
> compatibility.
>
> On Tue, Feb 4, 2020 at 3:14 PM Лев Киселев
> wrote:
> >
> > Thanks for answer, Andrey.
> > I'm confused about your first comment. Does it mean, that the only thing
> I
> > can do is to create new annotation?
> > Because, as you said, I must leave unchanged all related to old
> annotations
> > functionality.
> > So, I can't change implementation of *getDescription / getParameterName
> > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> > IgniteStandardMXBean* class,
> > and replace current annotations with new one.
> >
> > вт, 4 февр. 2020 г. в 14:28, Andrey Gura :
> >
> > > Hi, Lev!
> > >
> > > I think you understand task correctly.
> > >
> > > A couple of comments:
> > >
> > > - Existing annotations and related functionality must not be deleted
> > > because it is part of public API and we committed to have stable API
> > > between major releases.
> > > - I would suggest introduce @MXBeanParameter annotation instead of
> > > @MXBeanParameterInformation. Just shorten name.
> > >
> > > On Thu, Jan 30, 2020 at 8:53 PM Denis Magda wrote:
> > > >
> > > > Nikolay, Andrey,
> > > >
> > > > Would you be the best committers to help out here? You are already
> > > deeply involved in metrics development and can quickly suggest Lev how
> to
> > > proceed with this task.
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Thu, Jan 30, 2020 at 2:46 AM Лев Киселев <
> lev.kiselev.1...@gmail.com>
> > > wrote:
> > > >>
> > > >> Hello everyone, I have question about following task:
> > > >> [https://issues.apache.org/jira/browse/IGNITE-10698]
> > > >> Solution proposed in task description is seem to be logical.
> > > >> So, I need to every replace @MXBeanParametersNames and
> > > >> @MXBeanParametersDescriptions (everywhere, for uniformity) with
> > > something
> > > >> like:
> > > >> void methodName(@MXBeanParameterInformation(name = "name",
> description =
> > > >> "description") firstParameter, ...) {}.
> > > >> And, of course, need to change processing logic at
> > > >> getParameterName/getDescription methods from IgniteStandardMXBean.
> > > >> Do I understand correctly what needs to be done?
> > >
>
[DISCUSSION][IEP-39] Management API to cancel user provided tasks and queries.
Hello, Igniters.
I propose to create management API to dance user provided tasks and queries.
Below my proposal in the IEP [1] form.
Please, share your feedback.
Motivation:
Ignite provides many API to deploy and execute user-provided code on the
server nodes
inside the sam JVM as Ignite process runs.
We also has a many APIs that allocate many resources on the server nodes:
In case of some buggy code that consumes many system resources(CPU, RAM,
flood network) or heavy query
the whole cluster can becomes instable.
We should provide to the cluster administator the ability to stop any user
deployed task.
Description:
JMX beans to cancel listed tasks should be introduced:
* Compute task
* Service
* Continuous query
* Transactions
* Queries(SQL, Scan, Text)
In the scope of IEP-35 System view API introduced.
New API should use the same identifier that is used in corresponding System
View.
Public API changes:
* Cancel of Scan(text) queries should work similar to the SQL.
* Cancel of compute task should invoke `ComputeTaskInternalFuture#cancel`
for the corresponding task.
* Cancel of service should be the same as `IgniteServices#cancel`
* Cancel of transaction should works the same as in
`TransactionMXBean#getActiveTransaction(kill=true)`.
* Cancel of a continuous query should work the same as `QueryCursor#close`
execution.
```
QueryMXBean {
boolean cancelSQL(Long queryId); //Already implemented in IGNITE-4436.
boolean cancelScan(Long queryId);
boolean cancelText(Long queryId);
}
ComputeMXBean {
boolean cancel(IgniteUuid id);
}
ServiceMXBean {
boolean cancel(IgniteUuid id);
}
TransactionMXBean {
boolean cancel(IgniteUuid xid);
}
interface ContinuousQueryMXBean {
boolean cancel(UUID id);
}
```
[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=145724615
> 17 янв. 2020 г., в 22:43, Николай Ижиков написал(а):
>
> Hello, Andrey.
>
> Thanks for positive feedback.
> Appreciate it.
>
>> we can't cancel service or service's method
>
> I understand it.
> Seems, we have several obvious options here:
>
> * Try to do it and if fail answer to the user: «I’ve tried and fail. Sorry»
> * Kill hanging thread.
>
>> This invocations not register anywhere and can't be tracked and killed.
>
> Yes.
> So we should invent tracking for those invocation to be able to kill it :)
>
>
>> 17 янв. 2020 г., в 21:03, Andrey Gura написал(а):
>>
>> It would be grate.
>>
>> Only one comment: we can't cancel service or service's method
>> execution because service grid API doesn't imply any requirement to
>> service implementation. So if user do something like infinite cycle or
>> blocking but non-interruptible operation it's impossible to interrupt
>> it. Also service method invocation itself isn't cluster wide or local
>> node specific operation. This invocations not register anywhere and
>> can't be tracked and killed.
>>
>> Unfortunately, the same is valid for some jobs in sense of infinite or
>> non-interruptible operations.
>>
>> On Wed, Jan 15, 2020 at 9:30 PM Николай Ижиков wrote:
>>>
>>> Hello, Igniters.
>>>
>>> As you may know, we put a lot of effort to improve Ignite metric and
>>> diagnostic API.
>>> We have created the following API:
>>> * Metric manager
>>> * System view manager
>>> As far as I know, we would have tracing capabilities soon.
>>>
>>> I think it's time to take the next step.
>>> We should provide to the administrator the API to cancel(stop, kill)
>>> arbitrary user started computation.
>>>
>>> Right now we have API to do it for:
>>> * transactions `TransactionsMXBean#getActiveTransactions`.
>>> * SQL queries: `KILL QUERY` sql command and visor task -
>>> `VisorQueryCancelTask`
>>>
>>> Please, note, these features are implemented via different API.
>>>
>>> I think we should introduce uniform Cancel API for the following
>>> computations:
>>>
>>> * service.
>>> * specific service method execution.
>>> * compute job(compute task).
>>> * query(scan, continuous, text).
>>>
>>> We should also rework kill transaction and kill SQL queries API to make
>>> them similar to each other.
>>> I have plans to write an IEP for it and implement it.
>>> What do you think?
>>>
>
Re: JAVADOC fails on local build. Should it be checked on TC?
The next bit I found out is that links like "https://hadoop.apache.org/docs/current/api/" are needed to have working links to external classes in javadocs (e.g. Hadoop ones). Unfortunately, we are not consistent with external links. E.g. references to JCache classes are rendered as plain text in our javadocs [1]. For now I suggest to drop Hadoop links from maven plugin configuration as well. If there are no objections I will prepare a PR. [1] https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheEntryProcessor.html вт, 4 февр. 2020 г. в 15:04, Ivan Pavlukhin : > > Alexey, > > Thank you for the hint. Actually my bet was the same. But currently I > have no understanding how > "https://hadoop.apache.org/docs/current/api/" tag is used > and is it used at all. > > вт, 4 февр. 2020 г. в 15:01, Alexey Goncharuk : > > > > Ivan, > > > > My guess would be that the link was included for the Hadoop Accelerator > > which we all agreed to remove [1]. I think it is safe to remove it from > > pom.xml completely. > > > > [1] > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: Contribution
MXBeanParametersNames and MXBeanParametersDescriptions could be marked
as deprecated if MXBeanParameter will be added.
On Tue, Feb 4, 2020 at 5:12 PM Ivan Pavlukhin wrote:
>
> Should we mark old annotations deprecated?
>
> вт, 4 февр. 2020 г. в 16:57, Andrey Gura :
> >
> > Lev,
> >
> > > I'm confused about your first comment. Does it mean, that the only thing I
> > can do is to create new annotation?
> >
> > Yes, it does. But in this case you can mark old annotations as
> > deprecated because new annotations are real alternative.
> >
> > > So, I can't change implementation of *getDescription / getParameterName
> > > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> > > IgniteStandardMXBean* class,
> > > and replace current annotations with new one.
> >
> > You have to support both old and new annotations due to a backward
> > compatibility.
> >
> > On Tue, Feb 4, 2020 at 3:14 PM Лев Киселев
> > wrote:
> > >
> > > Thanks for answer, Andrey.
> > > I'm confused about your first comment. Does it mean, that the only thing I
> > > can do is to create new annotation?
> > > Because, as you said, I must leave unchanged all related to old
> > > annotations
> > > functionality.
> > > So, I can't change implementation of *getDescription / getParameterName
> > > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> > > IgniteStandardMXBean* class,
> > > and replace current annotations with new one.
> > >
> > > вт, 4 февр. 2020 г. в 14:28, Andrey Gura :
> > >
> > > > Hi, Lev!
> > > >
> > > > I think you understand task correctly.
> > > >
> > > > A couple of comments:
> > > >
> > > > - Existing annotations and related functionality must not be deleted
> > > > because it is part of public API and we committed to have stable API
> > > > between major releases.
> > > > - I would suggest introduce @MXBeanParameter annotation instead of
> > > > @MXBeanParameterInformation. Just shorten name.
> > > >
> > > > On Thu, Jan 30, 2020 at 8:53 PM Denis Magda wrote:
> > > > >
> > > > > Nikolay, Andrey,
> > > > >
> > > > > Would you be the best committers to help out here? You are already
> > > > deeply involved in metrics development and can quickly suggest Lev how
> > > > to
> > > > proceed with this task.
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Thu, Jan 30, 2020 at 2:46 AM Лев Киселев
> > > > >
> > > > wrote:
> > > > >>
> > > > >> Hello everyone, I have question about following task:
> > > > >> [https://issues.apache.org/jira/browse/IGNITE-10698]
> > > > >> Solution proposed in task description is seem to be logical.
> > > > >> So, I need to every replace @MXBeanParametersNames and
> > > > >> @MXBeanParametersDescriptions (everywhere, for uniformity) with
> > > > something
> > > > >> like:
> > > > >> void methodName(@MXBeanParameterInformation(name = "name",
> > > > >> description =
> > > > >> "description") firstParameter, ...) {}.
> > > > >> And, of course, need to change processing logic at
> > > > >> getParameterName/getDescription methods from IgniteStandardMXBean.
> > > > >> Do I understand correctly what needs to be done?
> > > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
Re: Contribution
Should we mark old annotations deprecated?
вт, 4 февр. 2020 г. в 16:57, Andrey Gura :
>
> Lev,
>
> > I'm confused about your first comment. Does it mean, that the only thing I
> can do is to create new annotation?
>
> Yes, it does. But in this case you can mark old annotations as
> deprecated because new annotations are real alternative.
>
> > So, I can't change implementation of *getDescription / getParameterName
> > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> > IgniteStandardMXBean* class,
> > and replace current annotations with new one.
>
> You have to support both old and new annotations due to a backward
> compatibility.
>
> On Tue, Feb 4, 2020 at 3:14 PM Лев Киселев wrote:
> >
> > Thanks for answer, Andrey.
> > I'm confused about your first comment. Does it mean, that the only thing I
> > can do is to create new annotation?
> > Because, as you said, I must leave unchanged all related to old annotations
> > functionality.
> > So, I can't change implementation of *getDescription / getParameterName
> > (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> > IgniteStandardMXBean* class,
> > and replace current annotations with new one.
> >
> > вт, 4 февр. 2020 г. в 14:28, Andrey Gura :
> >
> > > Hi, Lev!
> > >
> > > I think you understand task correctly.
> > >
> > > A couple of comments:
> > >
> > > - Existing annotations and related functionality must not be deleted
> > > because it is part of public API and we committed to have stable API
> > > between major releases.
> > > - I would suggest introduce @MXBeanParameter annotation instead of
> > > @MXBeanParameterInformation. Just shorten name.
> > >
> > > On Thu, Jan 30, 2020 at 8:53 PM Denis Magda wrote:
> > > >
> > > > Nikolay, Andrey,
> > > >
> > > > Would you be the best committers to help out here? You are already
> > > deeply involved in metrics development and can quickly suggest Lev how to
> > > proceed with this task.
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Thu, Jan 30, 2020 at 2:46 AM Лев Киселев
> > > wrote:
> > > >>
> > > >> Hello everyone, I have question about following task:
> > > >> [https://issues.apache.org/jira/browse/IGNITE-10698]
> > > >> Solution proposed in task description is seem to be logical.
> > > >> So, I need to every replace @MXBeanParametersNames and
> > > >> @MXBeanParametersDescriptions (everywhere, for uniformity) with
> > > something
> > > >> like:
> > > >> void methodName(@MXBeanParameterInformation(name = "name", description
> > > >> =
> > > >> "description") firstParameter, ...) {}.
> > > >> And, of course, need to change processing logic at
> > > >> getParameterName/getDescription methods from IgniteStandardMXBean.
> > > >> Do I understand correctly what needs to be done?
> > >
--
Best regards,
Ivan Pavlukhin
[jira] [Created] (IGNITE-12620) Calcite integration. Index Nested Loop Join/Hash Join
Igor Seliverstov created IGNITE-12620: - Summary: Calcite integration. Index Nested Loop Join/Hash Join Key: IGNITE-12620 URL: https://issues.apache.org/jira/browse/IGNITE-12620 Project: Ignite Issue Type: New Feature Reporter: Igor Seliverstov We may implement the feature the next way: # For each row from left node consume whole dataset from right node # Pass join condition as an argument of request() method of the right node # In case a data source at right is an index scan ## If there is no cursor opened - create a new cursor using bounds from request ## If there is an existing cursor - just check the cursor was opened using the same condition as passed one. ## After the right node signals EOD consume a next row from left and repeat from p 2. # In case a data source at right is a table scan with huge amount of rows ## If there is no cursor opened - create a new cursor ignoring passed condition ## If there is an existing cursor - just check the cursor was opened using the same condition as passed one. ## After the right node signals EOD consume a next row from left and repeat from p 2. # In case a data source at right is remote / is a table scan with small number of rows/ is a filter node with good enough selectivity ## Create a hash index for a data source ## If there is no cursor opened - create a new cursor using bounds from request ## If there is an existing cursor - just check the cursor was opened using the same condition as passed one. ## After the right node signals EOD consume a next row from left and repeat from p 2. Consider implementation specifics at optimization time, choose the cheapest variant -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Contribution
Lev,
> I'm confused about your first comment. Does it mean, that the only thing I
can do is to create new annotation?
Yes, it does. But in this case you can mark old annotations as
deprecated because new annotations are real alternative.
> So, I can't change implementation of *getDescription / getParameterName
> (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> IgniteStandardMXBean* class,
> and replace current annotations with new one.
You have to support both old and new annotations due to a backward
compatibility.
On Tue, Feb 4, 2020 at 3:14 PM Лев Киселев wrote:
>
> Thanks for answer, Andrey.
> I'm confused about your first comment. Does it mean, that the only thing I
> can do is to create new annotation?
> Because, as you said, I must leave unchanged all related to old annotations
> functionality.
> So, I can't change implementation of *getDescription / getParameterName
> (MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
> IgniteStandardMXBean* class,
> and replace current annotations with new one.
>
> вт, 4 февр. 2020 г. в 14:28, Andrey Gura :
>
> > Hi, Lev!
> >
> > I think you understand task correctly.
> >
> > A couple of comments:
> >
> > - Existing annotations and related functionality must not be deleted
> > because it is part of public API and we committed to have stable API
> > between major releases.
> > - I would suggest introduce @MXBeanParameter annotation instead of
> > @MXBeanParameterInformation. Just shorten name.
> >
> > On Thu, Jan 30, 2020 at 8:53 PM Denis Magda wrote:
> > >
> > > Nikolay, Andrey,
> > >
> > > Would you be the best committers to help out here? You are already
> > deeply involved in metrics development and can quickly suggest Lev how to
> > proceed with this task.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Jan 30, 2020 at 2:46 AM Лев Киселев
> > wrote:
> > >>
> > >> Hello everyone, I have question about following task:
> > >> [https://issues.apache.org/jira/browse/IGNITE-10698]
> > >> Solution proposed in task description is seem to be logical.
> > >> So, I need to every replace @MXBeanParametersNames and
> > >> @MXBeanParametersDescriptions (everywhere, for uniformity) with
> > something
> > >> like:
> > >> void methodName(@MXBeanParameterInformation(name = "name", description =
> > >> "description") firstParameter, ...) {}.
> > >> And, of course, need to change processing logic at
> > >> getParameterName/getDescription methods from IgniteStandardMXBean.
> > >> Do I understand correctly what needs to be done?
> >
Re: [IGNITE-12582] Configuration by property
Ivan, Thanks. I have answered. On Tue, 4 Feb 2020 at 15:03, Ivan Pavlukhin wrote: > Sergey, > > Thank you for your efforts! I left some comments in the ticket [1]. > > [1] https://issues.apache.org/jira/browse/IGNITE-12582 > > сб, 1 февр. 2020 г. в 18:35, Sergey Chernolyas < > sergey.chernol...@gmail.com>: > > > > Hi Ivan! > > > > https://github.com/apache/ignite/pull/7347 . It is link to PR. > Criticism > > is welcome :-) > > > > On Sat, 1 Feb 2020 at 18:15, Ivan Pavlukhin wrote: > > > > > Sergey, > > > > > > Could you please share your prototype, I would like to have a look. > > > > > > сб, 1 февр. 2020 г. в 16:59, schernolyas >: > > > > > > > > Hi Ivan ! > > > > > > > > Yes, I have the prototype. Right now, main question is can we remove > > > spring > > > > data modules into project with extensions for Ignite. > > > > > > > > > > > > > > > > -- > > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > > > > > -- > > - > > > > With best regards, Sergey Chernolyas > > > > -- > Best regards, > Ivan Pavlukhin > -- - With best regards, Sergey Chernolyas
[jira] [Created] (IGNITE-12619) Extend test coverage of [IGNITE-11056] SQL: Create a view with list of existing indexes
Ivan Pavlukhin created IGNITE-12619: --- Summary: Extend test coverage of [IGNITE-11056] SQL: Create a view with list of existing indexes Key: IGNITE-12619 URL: https://issues.apache.org/jira/browse/IGNITE-12619 Project: Ignite Issue Type: Bug Components: sql Reporter: Ivan Pavlukhin Assignee: Ivan Pavlukhin Fix For: 2.9 Extend SQL INDEXES system view test coverage. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Contribution
Thanks for answer, Andrey.
I'm confused about your first comment. Does it mean, that the only thing I
can do is to create new annotation?
Because, as you said, I must leave unchanged all related to old annotations
functionality.
So, I can't change implementation of *getDescription / getParameterName
(MBeanOperationInfo op, MBeanParameterInfo param, int seq)* methods from*
IgniteStandardMXBean* class,
and replace current annotations with new one.
вт, 4 февр. 2020 г. в 14:28, Andrey Gura :
> Hi, Lev!
>
> I think you understand task correctly.
>
> A couple of comments:
>
> - Existing annotations and related functionality must not be deleted
> because it is part of public API and we committed to have stable API
> between major releases.
> - I would suggest introduce @MXBeanParameter annotation instead of
> @MXBeanParameterInformation. Just shorten name.
>
> On Thu, Jan 30, 2020 at 8:53 PM Denis Magda wrote:
> >
> > Nikolay, Andrey,
> >
> > Would you be the best committers to help out here? You are already
> deeply involved in metrics development and can quickly suggest Lev how to
> proceed with this task.
> >
> > -
> > Denis
> >
> >
> > On Thu, Jan 30, 2020 at 2:46 AM Лев Киселев
> wrote:
> >>
> >> Hello everyone, I have question about following task:
> >> [https://issues.apache.org/jira/browse/IGNITE-10698]
> >> Solution proposed in task description is seem to be logical.
> >> So, I need to every replace @MXBeanParametersNames and
> >> @MXBeanParametersDescriptions (everywhere, for uniformity) with
> something
> >> like:
> >> void methodName(@MXBeanParameterInformation(name = "name", description =
> >> "description") firstParameter, ...) {}.
> >> And, of course, need to change processing logic at
> >> getParameterName/getDescription methods from IgniteStandardMXBean.
> >> Do I understand correctly what needs to be done?
>
Re: JAVADOC fails on local build. Should it be checked on TC?
Alexey, Thank you for the hint. Actually my bet was the same. But currently I have no understanding how "https://hadoop.apache.org/docs/current/api/" tag is used and is it used at all. вт, 4 февр. 2020 г. в 15:01, Alexey Goncharuk : > > Ivan, > > My guess would be that the link was included for the Hadoop Accelerator > which we all agreed to remove [1]. I think it is safe to remove it from > pom.xml completely. > > [1] > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html -- Best regards, Ivan Pavlukhin
Re: [IGNITE-12582] Configuration by property
Sergey, Thank you for your efforts! I left some comments in the ticket [1]. [1] https://issues.apache.org/jira/browse/IGNITE-12582 сб, 1 февр. 2020 г. в 18:35, Sergey Chernolyas : > > Hi Ivan! > > https://github.com/apache/ignite/pull/7347 . It is link to PR. Criticism > is welcome :-) > > On Sat, 1 Feb 2020 at 18:15, Ivan Pavlukhin wrote: > > > Sergey, > > > > Could you please share your prototype, I would like to have a look. > > > > сб, 1 февр. 2020 г. в 16:59, schernolyas : > > > > > > Hi Ivan ! > > > > > > Yes, I have the prototype. Right now, main question is can we remove > > spring > > > data modules into project with extensions for Ignite. > > > > > > > > > > > > -- > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > > > > -- > - > > With best regards, Sergey Chernolyas -- Best regards, Ivan Pavlukhin
Re: JAVADOC fails on local build. Should it be checked on TC?
Ivan, My guess would be that the link was included for the Hadoop Accelerator which we all agreed to remove [1]. I think it is safe to remove it from pom.xml completely. [1] http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html
Re: Forbid mixed cache groups with both atomic and transactional caches
+1 to avoid mixed tx-atomic cache groups. On the other side, doing atomic ops inside a tx have no relation to described reasons and is acceptable for me. вт, 4 февр. 2020 г. в 14:40, Alexey Goncharuk : > I support this change. While this has no much difference on the storage > level, the protocols are indeed are very different and thus should be > separated. > > вт, 4 февр. 2020 г. в 14:36, Anton Vinogradov : > > > Seems, we already started the separation by atomic operations restriction > > inside the transactions [1]. > > See no reason to allow mixes in this case. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2313 > > > > On Tue, Feb 4, 2020 at 2:28 PM Ivan Rakov wrote: > > > > > Igniters, > > > > > > Apparently it's possible in Ignite to configure a cache group with both > > > ATOMIC and TRANSACTIONAL caches. > > > Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* > tests. > > > In my opinion, it would be better to remove such possibility from the > > > product. There are several reasons: > > > > > > 1) The original idea of grouping caches was optimizing storage overhead > > and > > > PME time by joining data of similar caches into the same partitions. > > ATOMIC > > > and TRANSACTIONAL caches provide different guarantees and are designed > > for > > > different use cases, thus they can hardly be called "similar". > > > > > > 2) Diving deeper: synchronization protocols and possible reasons for > > > primary-backup divergences are conceptually different for ATOMIC and > > > TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery > > protocol > > > allows to recover consistency if any participating node will fail, but > > for > > > ATOMIC caches there's possible scenario with failure of primary node > > where > > > neither of backups will contain the most recent state of the data. > > Example: > > > one backup have received updates 1, 3, 5 while another have received > 2, 4 > > > (which is possible due to message reordering), and even tracking > counters > > > [1] won't restore the consistency. The problem is that we can't > > distinguish > > > what kind of conflict we have faced in case update counters have > diverged > > > in a mixed group. > > > > > > 3) Mixed groups are poorly tested. I can't find any tests except a > couple > > > of smoke tests in IgniteCacheGroupsTest. We can't be sure that > different > > > synchronization protocols will work correctly for such configurations, > > > especially under load and with a variety of dependent configuration > > > parameters. > > > > > > 4) I have never heard of any feedback on mixed groups. I have asked > > > different people on this and no one recalled any attempts to configure > > such > > > groups. I believe that in fact no one has ever tried to do it. > > > > > > Please let me know if you are aware of any cases where mixed groups are > > > used or reasons to keep them. Otherwise I'll create a ticket to > prohibit > > > mixed configurations. > > > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-11797 > > > > > > -- > > > Best Regards, > > > Ivan Rakov > > > > > > -- Best regards, Alexei Scherbakov
Re: Forbid mixed cache groups with both atomic and transactional caches
Hello, I absolutely agree with the proposed restriction. Additionally, this fact should be clearly stated on the documentation page related to cache groups https://apacheignite.readme.io/docs/cache-groups Thanks, S. вт, 4 февр. 2020 г. в 14:40, Alexey Goncharuk : > I support this change. While this has no much difference on the storage > level, the protocols are indeed are very different and thus should be > separated. > > вт, 4 февр. 2020 г. в 14:36, Anton Vinogradov : > > > Seems, we already started the separation by atomic operations restriction > > inside the transactions [1]. > > See no reason to allow mixes in this case. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2313 > > > > On Tue, Feb 4, 2020 at 2:28 PM Ivan Rakov wrote: > > > > > Igniters, > > > > > > Apparently it's possible in Ignite to configure a cache group with both > > > ATOMIC and TRANSACTIONAL caches. > > > Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* > tests. > > > In my opinion, it would be better to remove such possibility from the > > > product. There are several reasons: > > > > > > 1) The original idea of grouping caches was optimizing storage overhead > > and > > > PME time by joining data of similar caches into the same partitions. > > ATOMIC > > > and TRANSACTIONAL caches provide different guarantees and are designed > > for > > > different use cases, thus they can hardly be called "similar". > > > > > > 2) Diving deeper: synchronization protocols and possible reasons for > > > primary-backup divergences are conceptually different for ATOMIC and > > > TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery > > protocol > > > allows to recover consistency if any participating node will fail, but > > for > > > ATOMIC caches there's possible scenario with failure of primary node > > where > > > neither of backups will contain the most recent state of the data. > > Example: > > > one backup have received updates 1, 3, 5 while another have received > 2, 4 > > > (which is possible due to message reordering), and even tracking > counters > > > [1] won't restore the consistency. The problem is that we can't > > distinguish > > > what kind of conflict we have faced in case update counters have > diverged > > > in a mixed group. > > > > > > 3) Mixed groups are poorly tested. I can't find any tests except a > couple > > > of smoke tests in IgniteCacheGroupsTest. We can't be sure that > different > > > synchronization protocols will work correctly for such configurations, > > > especially under load and with a variety of dependent configuration > > > parameters. > > > > > > 4) I have never heard of any feedback on mixed groups. I have asked > > > different people on this and no one recalled any attempts to configure > > such > > > groups. I believe that in fact no one has ever tried to do it. > > > > > > Please let me know if you are aware of any cases where mixed groups are > > > used or reasons to keep them. Otherwise I'll create a ticket to > prohibit > > > mixed configurations. > > > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-11797 > > > > > > -- > > > Best Regards, > > > Ivan Rakov > > > > > >
Re: Forbid mixed cache groups with both atomic and transactional caches
I support this change. While this has no much difference on the storage level, the protocols are indeed are very different and thus should be separated. вт, 4 февр. 2020 г. в 14:36, Anton Vinogradov : > Seems, we already started the separation by atomic operations restriction > inside the transactions [1]. > See no reason to allow mixes in this case. > > [1] https://issues.apache.org/jira/browse/IGNITE-2313 > > On Tue, Feb 4, 2020 at 2:28 PM Ivan Rakov wrote: > > > Igniters, > > > > Apparently it's possible in Ignite to configure a cache group with both > > ATOMIC and TRANSACTIONAL caches. > > Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests. > > In my opinion, it would be better to remove such possibility from the > > product. There are several reasons: > > > > 1) The original idea of grouping caches was optimizing storage overhead > and > > PME time by joining data of similar caches into the same partitions. > ATOMIC > > and TRANSACTIONAL caches provide different guarantees and are designed > for > > different use cases, thus they can hardly be called "similar". > > > > 2) Diving deeper: synchronization protocols and possible reasons for > > primary-backup divergences are conceptually different for ATOMIC and > > TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery > protocol > > allows to recover consistency if any participating node will fail, but > for > > ATOMIC caches there's possible scenario with failure of primary node > where > > neither of backups will contain the most recent state of the data. > Example: > > one backup have received updates 1, 3, 5 while another have received 2, 4 > > (which is possible due to message reordering), and even tracking counters > > [1] won't restore the consistency. The problem is that we can't > distinguish > > what kind of conflict we have faced in case update counters have diverged > > in a mixed group. > > > > 3) Mixed groups are poorly tested. I can't find any tests except a couple > > of smoke tests in IgniteCacheGroupsTest. We can't be sure that different > > synchronization protocols will work correctly for such configurations, > > especially under load and with a variety of dependent configuration > > parameters. > > > > 4) I have never heard of any feedback on mixed groups. I have asked > > different people on this and no one recalled any attempts to configure > such > > groups. I believe that in fact no one has ever tried to do it. > > > > Please let me know if you are aware of any cases where mixed groups are > > used or reasons to keep them. Otherwise I'll create a ticket to prohibit > > mixed configurations. > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-11797 > > > > -- > > Best Regards, > > Ivan Rakov > > >
Re: Forbid mixed cache groups with both atomic and transactional caches
Anton, Indeed, that's +1 point for forbidding mixed configurations. On Tue, Feb 4, 2020 at 2:36 PM Anton Vinogradov wrote: > Seems, we already started the separation by atomic operations restriction > inside the transactions [1]. > See no reason to allow mixes in this case. > > [1] https://issues.apache.org/jira/browse/IGNITE-2313 > > On Tue, Feb 4, 2020 at 2:28 PM Ivan Rakov wrote: > > > Igniters, > > > > Apparently it's possible in Ignite to configure a cache group with both > > ATOMIC and TRANSACTIONAL caches. > > Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests. > > In my opinion, it would be better to remove such possibility from the > > product. There are several reasons: > > > > 1) The original idea of grouping caches was optimizing storage overhead > and > > PME time by joining data of similar caches into the same partitions. > ATOMIC > > and TRANSACTIONAL caches provide different guarantees and are designed > for > > different use cases, thus they can hardly be called "similar". > > > > 2) Diving deeper: synchronization protocols and possible reasons for > > primary-backup divergences are conceptually different for ATOMIC and > > TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery > protocol > > allows to recover consistency if any participating node will fail, but > for > > ATOMIC caches there's possible scenario with failure of primary node > where > > neither of backups will contain the most recent state of the data. > Example: > > one backup have received updates 1, 3, 5 while another have received 2, 4 > > (which is possible due to message reordering), and even tracking counters > > [1] won't restore the consistency. The problem is that we can't > distinguish > > what kind of conflict we have faced in case update counters have diverged > > in a mixed group. > > > > 3) Mixed groups are poorly tested. I can't find any tests except a couple > > of smoke tests in IgniteCacheGroupsTest. We can't be sure that different > > synchronization protocols will work correctly for such configurations, > > especially under load and with a variety of dependent configuration > > parameters. > > > > 4) I have never heard of any feedback on mixed groups. I have asked > > different people on this and no one recalled any attempts to configure > such > > groups. I believe that in fact no one has ever tried to do it. > > > > Please let me know if you are aware of any cases where mixed groups are > > used or reasons to keep them. Otherwise I'll create a ticket to prohibit > > mixed configurations. > > > > [1]: https://issues.apache.org/jira/browse/IGNITE-11797 > > > > -- > > Best Regards, > > Ivan Rakov > > >
Re: Forbid mixed cache groups with both atomic and transactional caches
Seems, we already started the separation by atomic operations restriction inside the transactions [1]. See no reason to allow mixes in this case. [1] https://issues.apache.org/jira/browse/IGNITE-2313 On Tue, Feb 4, 2020 at 2:28 PM Ivan Rakov wrote: > Igniters, > > Apparently it's possible in Ignite to configure a cache group with both > ATOMIC and TRANSACTIONAL caches. > Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests. > In my opinion, it would be better to remove such possibility from the > product. There are several reasons: > > 1) The original idea of grouping caches was optimizing storage overhead and > PME time by joining data of similar caches into the same partitions. ATOMIC > and TRANSACTIONAL caches provide different guarantees and are designed for > different use cases, thus they can hardly be called "similar". > > 2) Diving deeper: synchronization protocols and possible reasons for > primary-backup divergences are conceptually different for ATOMIC and > TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery protocol > allows to recover consistency if any participating node will fail, but for > ATOMIC caches there's possible scenario with failure of primary node where > neither of backups will contain the most recent state of the data. Example: > one backup have received updates 1, 3, 5 while another have received 2, 4 > (which is possible due to message reordering), and even tracking counters > [1] won't restore the consistency. The problem is that we can't distinguish > what kind of conflict we have faced in case update counters have diverged > in a mixed group. > > 3) Mixed groups are poorly tested. I can't find any tests except a couple > of smoke tests in IgniteCacheGroupsTest. We can't be sure that different > synchronization protocols will work correctly for such configurations, > especially under load and with a variety of dependent configuration > parameters. > > 4) I have never heard of any feedback on mixed groups. I have asked > different people on this and no one recalled any attempts to configure such > groups. I believe that in fact no one has ever tried to do it. > > Please let me know if you are aware of any cases where mixed groups are > used or reasons to keep them. Otherwise I'll create a ticket to prohibit > mixed configurations. > > [1]: https://issues.apache.org/jira/browse/IGNITE-11797 > > -- > Best Regards, > Ivan Rakov >
Re: JAVADOC fails on local build. Should it be checked on TC?
I continued my experiments with javadocs and a next hurdle was an external link to Hadoop documentation https://hadoop.apache.org/docs/current/api/ . Javadoc build job scans logs for javadoc-specific warnings and it founds any it fails build. But there were many failures in the past in a form: "javadoc: warning - Error fetching URL: https://hadoop.apache.org/docs/current/api/; Consequently we ignore such lines. But a check we do seems unfair as we currently have malformed javadocs. Currently we use a following regexp to find javadoc warnings: "^\[WARNING\] javadoc: warning - (?!Error fetching URL).*$" I suppose a proper strategy is to search for "Javadoc Warnings" substring. But we must deal with a Hadoop external link. I made an experiment and removed "https://hadoop.apache.org/docs/current/api/" tag from parent/pom.xml and fixed all other existed javadoc warnings. I received a green build result [1] (in an experimental build job). Do we really need this link to Hadoop javadocs? What do you think? [1] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_ExperimentJavadoc?branch=pull%2F7356%2Fhead=overview=builds пт, 24 янв. 2020 г. в 16:06, Ivan Pavlukhin : > > Actually, I got stuck with a question what do modules like > ignite-hibernate mean from a javadoc standpoint. It seems that modules > sharing same packages and classes cannot be aggregated into valid > javadoc bundle. And actually this problem should be solved as well > (modularization?). But I suppose that it is closer to a release > process rather than daily TC check. > > So, my current understanding is: > 1. Javadoc job should be fixed to reveal problems (with code links). > 2. "javadoc" profile is not needed for daily TC check, but it > highlights some release-related problems. > > пт, 24 янв. 2020 г. в 15:44, Maxim Muzafarov : > > > > Folks, > > > > > > It's true that my simple assumption (to include Javadoc profile) will > > not work by reasons mentioned by Petr. I've tested it under a new > > experimental suite [1] which I've created earlier. We will always have > > such warnings due to we have multiple modules with the same package > > declarations (e.g. ignite-hibernate_5.1, hibernate_4.2, hibernate_5.3 > > modules and the same for others like spring-data): > > > > [WARNING] Javadoc: warning - Multiple sources of package comments > > found for package "org.apache.ignite.cache.store.hibernate" > > > > So, if we build all modules it's not correct to fail the whole JavaDoc > > suite by existing such warnings in the suite logs. > > > > > > I'm still thinking of how can we handle all such cases and I don't > > have an easy solution for now. At least we can do checks with two > > steps: > > 1. build with Javadoc profile ignore warnings (fail on an error) > > 2. build and check that there is no warnings for ignite-core module > > > > Thoughts? > > > > > > [1] > > https://ci.ignite.apache.org/viewLog.html?buildId=4927599=IgniteTests24Java8_JavadocExtended > > > > > > > > On Fri, 24 Jan 2020 at 13:47, Ivan Pavlukhin wrote: > > > > > > Petr, > > > > > > Thank you for that detail! Unfortunately I do not know how to fix it > > > easily. > > > > > > Also I found that even without "javadoc" profile we miss some javadoc > > > problems. I believe that it can be fixed by updating post-build > > > javadoc checking condition. See below what problems do we miss (and > > > there are a bit more in full logs): > > > [05:08:51]W: [Step 2/2] [WARNING] Javadoc Warnings > > > [05:08:51]W: [Step 2/2] [WARNING] > > > /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheContext.java:2351: > > > warning - Tag @link: reference not found: > > > GridDistributedCacheAdapter.GlobalRemoveAllJob > > > [05:08:51]W: [Step 2/2] [WARNING] > > > /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheContext.java:2351: > > > warning - Tag @link: reference not found: > > > GridDistributedCacheAdapter.GlobalRemoveAllJob > > > > > > пт, 24 янв. 2020 г. в 11:56, Petr Ivanov : > > > > > > > > We could not because of multiple hibernate modules, what is considered > > > > from the javadoc's point as warning, that always fails test. > > > > > > > > > > > > > On 24 Jan 2020, at 11:02, Ivan Pavlukhin wrote: > > > > > > > > > > Maxim, > > > > > > > > > > I suppose we can try to enable "javadoc" profile for a corresponding > > > > > job. Are there any objections to do so? > > > > > > > > > > пн, 20 янв. 2020 г. в 13:25, Ivan Pavlukhin : > > > > >> > > > > >> Maxim, > > > > >> > > > > >> From the first glance it seems that "javadoc" profile was really > > > > >> missed. Are there any other problems except springdata22? If no then > > > > >> we can add the profile. Also it is interesting how it influence on > > > > >> execution time? > > > > >> > > > > >> пн, 13 янв. 2020 г. в 16:53, Maxim Muzafarov : > > > > >>> > > > > >>> Igniters,
Re: Contribution
Hi, Lev!
I think you understand task correctly.
A couple of comments:
- Existing annotations and related functionality must not be deleted
because it is part of public API and we committed to have stable API
between major releases.
- I would suggest introduce @MXBeanParameter annotation instead of
@MXBeanParameterInformation. Just shorten name.
On Thu, Jan 30, 2020 at 8:53 PM Denis Magda wrote:
>
> Nikolay, Andrey,
>
> Would you be the best committers to help out here? You are already deeply
> involved in metrics development and can quickly suggest Lev how to proceed
> with this task.
>
> -
> Denis
>
>
> On Thu, Jan 30, 2020 at 2:46 AM Лев Киселев
> wrote:
>>
>> Hello everyone, I have question about following task:
>> [https://issues.apache.org/jira/browse/IGNITE-10698]
>> Solution proposed in task description is seem to be logical.
>> So, I need to every replace @MXBeanParametersNames and
>> @MXBeanParametersDescriptions (everywhere, for uniformity) with something
>> like:
>> void methodName(@MXBeanParameterInformation(name = "name", description =
>> "description") firstParameter, ...) {}.
>> And, of course, need to change processing logic at
>> getParameterName/getDescription methods from IgniteStandardMXBean.
>> Do I understand correctly what needs to be done?
Forbid mixed cache groups with both atomic and transactional caches
Igniters, Apparently it's possible in Ignite to configure a cache group with both ATOMIC and TRANSACTIONAL caches. Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests. In my opinion, it would be better to remove such possibility from the product. There are several reasons: 1) The original idea of grouping caches was optimizing storage overhead and PME time by joining data of similar caches into the same partitions. ATOMIC and TRANSACTIONAL caches provide different guarantees and are designed for different use cases, thus they can hardly be called "similar". 2) Diving deeper: synchronization protocols and possible reasons for primary-backup divergences are conceptually different for ATOMIC and TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery protocol allows to recover consistency if any participating node will fail, but for ATOMIC caches there's possible scenario with failure of primary node where neither of backups will contain the most recent state of the data. Example: one backup have received updates 1, 3, 5 while another have received 2, 4 (which is possible due to message reordering), and even tracking counters [1] won't restore the consistency. The problem is that we can't distinguish what kind of conflict we have faced in case update counters have diverged in a mixed group. 3) Mixed groups are poorly tested. I can't find any tests except a couple of smoke tests in IgniteCacheGroupsTest. We can't be sure that different synchronization protocols will work correctly for such configurations, especially under load and with a variety of dependent configuration parameters. 4) I have never heard of any feedback on mixed groups. I have asked different people on this and no one recalled any attempts to configure such groups. I believe that in fact no one has ever tried to do it. Please let me know if you are aware of any cases where mixed groups are used or reasons to keep them. Otherwise I'll create a ticket to prohibit mixed configurations. [1]: https://issues.apache.org/jira/browse/IGNITE-11797 -- Best Regards, Ivan Rakov
Re: Internal classes are exposed in public API
Folks, Let's start a vote? On Mon, 3 Feb 2020 at 15:05, Andrey Gura wrote: > > Just post here article from Oracle documentation "How and When To > Deprecate APIs" [1]. > > [1] > https://docs.oracle.com/javase/8/docs/technotes/guides/javadoc/deprecation/deprecation.html > > On Fri, Jan 31, 2020 at 10:44 AM Pavel Tupitsyn wrote: > > > > Agree with Andrey, let's remove deprecation for now and unblock the release. > > > > On Thu, Jan 30, 2020 at 10:23 PM Andrey Gura wrote: > > > > > I'll just repeat one thought with some changes. > > > > > > There are at least two groups of people in this discussion. One group > > > sure that new API is complete and production ready while other group > > > disagree with it. Obviously we can't reach consensus about it right > > > now. But we can do it in the future. > > > At present we just can't deprecate existing API and mark new API as > > > experimental at the same time. So we must remove deprecation until the > > > consensus is reached. > > > > > > Also there is the third group of people. This people aren't related > > > with the API, they may be don't need the API and they wait for bug > > > fixes and other features. It is very easy to satisfy third group: just > > > cut off what caused the release blocking. But it is much easier to > > > remove @deprecated annotations. > > > > > > On Thu, Jan 30, 2020 at 8:54 PM Nikolay Izhikov > > > wrote: > > > > > > > > Alexey. > > > > > > > > I answered to your examples and issues you provide. > > > > But, it seems the discussion of the API and the Java code itself is not > > > the goal of this thread anymore. > > > > > > > > > Should we provide a way to know the number of metrics and registries > > > in advance? > > > > > > > > No. > > > > If you think this is the real use-as let’s add `size` methods. > > > > It will be the simple API *extension*. > > > > > > > > > There is no separation on public and internal metrics > > > > > > > > Any metric can be changed(removed) in any time. > > > > But we will try not to do it unless we have a strong reason. > > > > > > > > > Will we allow users to register their own metrics? > > > > > > > > No. > > > > > > > > > It's still not clear how a user will map old interfaces and methods to > > > the new metric names. > > > > > > > > We should write this information in the deprecation message. > > > > > > > > > 30 янв. 2020 г., в 20:27, Alexey Goncharuk > > > > > > > > написал(а): > > > > > > > > > > Nikita, > > > > > > > > > > Disagree here. I already gave an example in this thread of how you > > > need to > > > > > peek into configuration in order to obtain an instance of exporter SPI > > > > > which may not necessarily be the type you need. That's why > > > IGNITE-12553 was > > > > > created in the first place. > > > > > > >
Re: Contribution. IGNITE-11663
Artem, I've answered to your question in apache slack. Better to discuss implementation details in ticket comments. пт, 31 янв. 2020 г. в 21:02, Denis Magda : > Alex Scherbakof, as the one who created the ticket, could you please help > Artem and share more details? > > - > Denis > > > On Fri, Jan 31, 2020 at 4:54 AM Artem Demchenko > wrote: > >> Hi, everyone! I have a question about the following task: >> https://issues.apache.org/jira/browse/IGNITE-11663. >> >> The description is not quite concrete, so I want to clarify requirements. >> >> Currently, there are >> “internal.processors.cache.persistence.wal.record.RecordTypes.java” class, >> which contains one class field “public static final >> Set DELTA_TYPE_SET = new HashSet<>();” and static >> block to fill set above with particular values. This class has only one >> usage. >> >> My PR https://github.com/apache/ignite/pull/7330 purpose to move all >> logic >> from “internal.processors.cache.persistence.wal.record.RecordTypes.java” to >> the only place where it used >> “internal.processors.cache.persistence.wal.SingleSegmentLogicalRecordsIterator.java”. >> >> Am I doing everything right or not? If not, what has to be done? >> >> >> -- >> Demchenko Artem. >> > -- Best regards, Alexei Scherbakov
