Re: Issue with replicated cache
Hi Prasad,
Can you please share logs from all nodes, so I can check what was happening
with a cluster before an incident? It would be great to see logs since
nodes start.
Thanks,
Evgenii
чт, 26 дек. 2019 г. в 11:42, Denis Magda :
> Let me loop in the Ignite dev list as long as I've not heard about such an
> issue. Personally, don't see any misconfiguration in your Ignite config.
>
> -
> Denis
>
>
> On Thu, Dec 26, 2019 at 10:17 AM Prasad Bhalerao <
> prasadbhalerao1...@gmail.com> wrote:
>
> > I used cache.remove(key) method to delete an entry from cache.
> >
> > Basically I was not getting the consistent result on subsequent API
> > calls with the same input.
> >
> > So I used grid gain console to query the cache. I executed the SQL on
> > single node at a time.
> > While doing this I found data only on node n1. But same entry was not
> > present on nodes n2,n3,n4.
> >
> > Thanks,
> > Prasad
> >
> >
> >
> >
> > On Thu 26 Dec, 2019, 11:09 PM Denis Magda >
> >> Hello Prasad,
> >>
> >> What APIs did you use to remove the entry from the cache and what method
> >> did you use to confirm that the entry still exists on some of the nodes?
> >>
> >> -
> >> Denis
> >>
> >>
> >> On Thu, Dec 26, 2019 at 8:54 AM Prasad Bhalerao <
> >> prasadbhalerao1...@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am using ignite 2.6.0 version and the time out settings are as
> follows.
> >>>
> >>> IgniteConfiguration cfg = new IgniteConfiguration();
> >>> cfg.setFailureDetectionTimeout(12);
> >>> cfg.setNetworkTimeout(1);
> >>> cfg.setClientFailureDetectionTimeout(12);
> >>>
> >>> I have 4 server nodes (n1,n2,n3,n4) and 6 client nodes. I am using a
> >>> replicated cache and cache configuration is as shown below.
> >>> As you can see write-through is false, read through is true and write
> >>> synchronization mode is FULL_SYNC.
> >>>
> >>> I got an issue, a network entry was removed from network cache but some
> >>> how it was removed from only 3 server nodes (n2,n3,n4). I was able to
> see
> >>> the network entry on node n1 consistently for a day(when it was
> removed).
> >>> So I checked the logs for any errors/warnings but I could not find any.
> >>> I did not see any segmentation issue in logs, looked like cluster was
> in
> >>> healthy state.
> >>> When I checked the cache after 2 days, I could not find that entry.
> >>> Cache was in a state as it was supposed to be. Servers were not
> stopped
> >>> and restarted during this whole time.
> >>>
> >>> Some how I am not able to reproduce this issue on dev env.
> >>>
> >>> Is there any way to investigate/debug this issue? Can someone please
> >>> advise?
> >>>
> >>> private CacheConfiguration networkCacheCfg() {
> >>> CacheConfiguration networkCacheCfg = new CacheConfiguration<>(
> CacheName.NETWORK_CACHE.name());
> >>> networkCacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> >>> networkCacheCfg.setWriteThrough(false);
> >>> networkCacheCfg.setReadThrough(true);
> >>> networkCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
> >>>
>
> networkCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
> >>> networkCacheCfg.setBackups(this.backupCount);
> >>> networkCacheCfg.setCacheMode(CacheMode.REPLICATED);
> >>> Factory storeFactory =
> FactoryBuilder.factoryOf(NetworkDataCacheLoader.class);
> >>> networkCacheCfg.setCacheStoreFactory(storeFactory);
> >>> networkCacheCfg.setIndexedTypes(DefaultDataAffinityKey.class,
> NetworkData.class);
> >>> networkCacheCfg.setSqlIndexMaxInlineSize(65);
> >>> RendezvousAffinityFunction affinityFunction = new
> RendezvousAffinityFunction();
> >>> affinityFunction.setExcludeNeighbors(true);
> >>> networkCacheCfg.setAffinity(affinityFunction);
> >>> networkCacheCfg.setStatisticsEnabled(true);
> >>>
> >>> return networkCacheCfg;
> >>> }
> >>>
> >>>
> >>>
> >>> Thanks,
> >>> PRasad
> >>>
> >>>
>
Re: SQL dialects supported by Calcite
Igor, When you are saying that we should not allow the dialect changing, are you referring to changes in runtime when a node is already up-and-running? Generally, it will be more than enough if a dialect can be set statically and globally -- the user selects the dialect for the entire cluster via a configuration parameter and starts the nodes after that. - Denis On Fri, Dec 27, 2019 at 7:05 AM Seliverstov Igor wrote: > Denis, > > > Is it true that Calcite can parse MySQL, Oracle > > Yes, it can. > > > Do I need to select a dialect globally or can it be set on a per-query > > level? > > Technically a parser, a validator, a planner, other components are created > and configured for each query call (because they are stateful). So you may > configure it per query, or hardcode a desired dialect, or get it from > configuration - it’s up to you, but we expect parser configuration to be > (more or less) static, it is a part of initial framework configuration. I > think we should not allow such parameters (as a dialect) changing. > > Regards, > Igor > > > 27 дек. 2019 г., в 07:47, Denis Magda написал(а): > > > > Igor S., Roman and others who involved in Calcite prototyping, > > > > Is it true that Calcite can parse MySQL, Oracle, ANSI-99 and other > dialects > > as suggested by this page? > > > https://calcite.apache.org/apidocs/org/apache/calcite/sql/validate/SqlConformanceEnum.html > > > > Do I need to select a dialect globally or can it be set on a per-query > > level? > > > > - > > Denis > >
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Ilya, no I see, thanks for the explanation. Agree with you, let's update
the versions of the dependencies to the latest.
-
Denis
On Thu, Dec 26, 2019 at 10:50 PM Ilya Kasnacheev
wrote:
> Hello!
>
> I have committed ignite-spring-data_2.2 to ignite-2.8.
>
> By bumping versisons I mean the following:
> 1.7.*7*
> 1.6.*4*
> 1.1.7.*2*
> 2.6.*5*
> 2.3.*0*
> 1.13.*14*.RELEASE
> 4.3.*18*.RELEASE
> 2.0.*9*.RELEASE
>
> 5.0.*8*.RELEASE
>
> All these libraries have maintenance release (such as our 2.7.*6*) and I
> think it would be beneficial to upgrade these dependencies to the latest
> maintenance version found in Maven Central.
> For example, there is spring.data-2.0 2.0.*14*.RELEASE.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 26 дек. 2019 г. в 19:32, Denis Magda :
>
> > A huge +1 for adding Spring Data related fixes/improvements. Ilya is
> right
> > that Spring Data related questions sparked last time due to missing
> support
> > of 2.2 version.
> >
> > Ilya, could you elaborate on what you mean under "bumping the versions"?
> Do
> > you suggest performing a straightforward upgrade of "ignite-spring-data"
> to
> > version 2.2 and introducing "ignite-spring-data-{old-version"} for the
> > previous versions? If it's so, I fully agree with the proposal.
> >
> > -
> > Denis
> >
> >
> > On Thu, Dec 26, 2019 at 4:52 AM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >
> > wrote:
> >
> > > Hello!
> > >
> > > I propose to add the following ticket to the scope:
> > > https://issues.apache.org/jira/browse/IGNITE-12259 (3 commits, be
> > careful
> > > with release version)
> > >
> > > Adding tickets to scope surely seems crazy now, but I will provide the
> > > following considerations:
> > > * This is Spring Data 2.2 integration, which we currently do not have,
> > > leading to lots of confused questions on stack overflow and mailing
> list.
> > > Spring Data is important to our public image since many people may
> learn
> > > about out project by starting with Spring Data.
> > >
> > > * It has zero code impact outside of its own module (just 2 POM file
> > > touched and that's all).
> > >
> > > * The core was ready since early November but, due to gmail quirk, we
> did
> > > not react to it in time.
> > >
> > > WDYT?
> > >
> > > Another semi-related question. *Should we bump our dependencies'
> versions
> > > before releasing 2.8?* I talk mainly about spring and hibernate
> > > dependencies. We could switch them to their latest maintenance versions
> > to
> > > avoid shipping default links to outdated packages.
> > >
> > > I think this is one of things that are very hard to do between
> releases,
> > so
> > > I think this dependencies bumping should be a part of a formal
> > > release/testing cycle, and then be backported to master.
> > >
> > > I could volunteer to do that myself, if we agree to merge these version
> > > upgrades to ignite-2.8 and then re-test.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > вт, 24 дек. 2019 г. в 13:22, Zhenya Stanilovsky
> > > > >:
> > >
> > > >
> > > > Igniters, i`l try to compare 2.8 release candidate vs 2.7.6,
> > > > last sha 2.8 was build from : 9d114f3137f92aebc2562a
> > > > i use yardstick benchmarks, 4 bare machine with: 2x Xeon X5570 96Gb
> > > 512GB
> > > > SSD 2048GB HDD 10GB/s
> > > > 1 for client (driver) and 3 for servers.
> > > > this mappings for graphs and real yardstick tests:
> > > >
> > > > atomic-put: IgnitePutBenchmark
> > > > sql-merge-query: IgniteSqlMergeQueryBenchmark
> > > > atomic-get: IgniteGetBenchmark
> > > > tx-get: IgniteGetTxBenchmark
> > > > tx-put: IgnitePutTxBenchmark
> > > > atomic-put-all-bs-10: IgnitePutAllBenchmark
> > > > tx-put-all-bs-10: IgnitePutAllTxBenchmark
> > > >
> > > > cacheMode — partitioned
> > > > CacheWriteSynchronizationMode.FULL_SYNC
> > > > 1 backup
> > > >
> > > > 1. wal = log_only 2. wal = none 3. persistence disabled.
> > > > Thanks Maxim for wiki page [1]
> > > >
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks
> > > >
> > > > do we need some bisect or other work here ?
> > > >
> > > > >
> > > > >
> > > > >--- Forwarded message ---
> > > > >From: "Maxim Muzafarov" < mmu...@apache.org >
> > > > >To: dev@ignite.apache.org
> > > > >Cc:
> > > > >Subject: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
> > > > >Date: Fri, 20 Sep 2019 14:44:31 +0300
> > > > >
> > > > >Igniters,
> > > > >
> > > > >
> > > > >It's almost a year has passed since the last major Apache Ignite 2.7
> > > > >has been released. We've accumulated a lot of performance
> improvements
> > > > >and a lot of new features which are waiting for their release date.
> > > > >Here is my list of the most interesting things from my point since
> the
> > > > >last major release:
> > > > >
> > > > >Service Grid,
> > > > >Monitoring,
> > > > >Recovery Read
> > > > >BLT
Re: Submit Ignite talk and present at O'Reilly Conference in Silicon Valley
Excellent, I've submitted an in-memory computing essentials talk there as well. I hope at least one of the talks is to be selected. Keeping fingers crossed. - Denis On Fri, Dec 27, 2019 at 10:41 AM Saikat Maitra wrote: > Hi, > > Thank you for sharing the link Denis. I have submitted a talk on Data > Streaming. > > Regards, > Saikat > > On Wed, Nov 27, 2019 at 5:25 PM Denis Magda wrote: > > > Igniters, > > > > There are many of us who are spreading the word about Ignite globally. > Does > > anybody have any technical talk of moderate and advanced complexity for > the > > O'Reilly Software Architecture conference? > > > https://conferences.oreilly.com/software-architecture/sa-ca/public/content > > > > The more we submit the bigger are the chances that a community > > representative will be selected for the conference. In case of selection, > > if you or your employer is not ready to sponsor the trip, then GridGain > > will cover associated travel costs (air tickets and a hotel for the > > conference days). > > > > > > - > > Denis > > >
Re: Submit Ignite talk and present at O'Reilly Conference in Silicon Valley
Hi, Thank you for sharing the link Denis. I have submitted a talk on Data Streaming. Regards, Saikat On Wed, Nov 27, 2019 at 5:25 PM Denis Magda wrote: > Igniters, > > There are many of us who are spreading the word about Ignite globally. Does > anybody have any technical talk of moderate and advanced complexity for the > O'Reilly Software Architecture conference? > https://conferences.oreilly.com/software-architecture/sa-ca/public/content > > The more we submit the bigger are the chances that a community > representative will be selected for the conference. In case of selection, > if you or your employer is not ready to sponsor the trip, then GridGain > will cover associated travel costs (air tickets and a hotel for the > conference days). > > > - > Denis >
Re: Contribution page & easy/moderate tickets for newcomers
Hi Denis, Yes, adding other form of contributions like help in answering user questions, updating documents and reviewing code changes will certainly help in learning and knowing the project better We can also add this link for reference for open issues waiting for review, this will help folks to learn on code styles which our community is following. https://cwiki.apache.org/confluence/display/IGNITE/Issues+waiting+for+review Regards, Saikat On Thu, Dec 26, 2019 at 10:22 PM Denis Magda wrote: > Hello Saikat, > > Thanks for checking the page and suggesting the improvement. I'll add the > link to the page. > > Also, I'm thinking to go for another round of changes by listing other > forms of contributions such as helping to answer users' questions, > reviewing changes or updating documentation. Like the way it's described by > Spark community: https://spark.apache.org/contributing.html > > - > Denis > > > On Thu, Dec 26, 2019 at 12:18 PM Saikat Maitra > wrote: > > > Hello Denis, > > > > Thank you for publishing and modifying the Contributions page. Many time > > folks have approached me over email, linkedin and offline about how they > > can contribute to Apache projects and I share this link and about my > > experience. I will continue to share this link going forward. > > > > I am thinking if it will be ok to add link to the apache getting started > > 101 page https://community.apache.org/gettingStarted/101.html in the > > Contributions page. > > > > Regards, > > Saikat > > > > > > > > > > > > > > > > On Sat, Dec 21, 2019 at 7:53 AM Denis Magda wrote: > > > > > Pavel, Alexey, thanks for checking and updates! > > > > > > I’ll send this page to a fellow editor for a review once it’s checked > by > > us > > > internally. Let’s wait a bit letting others join the thread. > > > > > > Denis > > > > > > On Saturday, December 21, 2019, Alexey Zinoviev < > zaleslaw@gmail.com> > > > wrote: > > > > > > > Cool, filters much more better than previous list of tasks inserted > > > > directly in HTML > > > > > > > > сб, 21 дек. 2019 г. в 11:31, Pavel Tupitsyn : > > > > > > > > > Hi Denis, > > > > > > > > > > I've reviewed the list of newbie tickets for .NET, added some more > > > there. > > > > > > > > > > Also I've updated the page a little bit: > > > > > > > > > > - Changed the wording of the first paragraph: "you need to sign up" > > -> > > > > > "please sign up" > > > > > To be honest, the mood on this whole page feels a bit off to me. > > > > > I'm not a native speaker though, maybe we should ask for help here > to > > > > make > > > > > the text more friendly. > > > > > > > > > > - Fixed the JIRA links > > > > > Many of them pointed to a specific issue instead along with the > > search > > > > > results > > > > > > > > > > Thanks, > > > > > Pavel > > > > > > > > > > On Sat, Dec 21, 2019 at 12:07 AM Denis Magda > > > wrote: > > > > > > > > > > > Ignite, > > > > > > > > > > > > I've done a major overhaul of our contribution page that promotes > > > > > > involvement in Ignite development: > > > > > > https://ignite.apache.org/community/contribute.html > > > > > > > > > > > > Many sections were simplified, some legacy stuff was removed and > > I'll > > > > > > appreciate if some of you check the page and advise further > > > > improvements. > > > > > > > > > > > > Apart from that, one of the sections (Pick a Ticket) lists of > easy > > > and > > > > > > moderate tickets > > > > > > < > > > > > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20% > > > > > 3D%20IGNITE%20AND%20labels%20in%20(newbie)%20and%20status%20%3D%20OPEN > > > > > > > > > > > > > for > > > > > > newcomers. I do believe that the list needs to be reviewed. Could > > > each > > > > of > > > > > > you recall any tickets of easy/moderate complexity that are still > > > > pending > > > > > > but will be great to implement? Put the "newbie" label for those > > and > > > > > > mention "component". I've already highlighted some of the > > components > > > on > > > > > the > > > > > > webpage (ML, SQL, platforms) and we can expand the list. > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > -- > > > - > > > Denis > > > > > >
[jira] [Created] (IGNITE-12510) In-memory page eviction may fail in case very large entries are stored in the cache
Ivan Rakov created IGNITE-12510:
---
Summary: In-memory page eviction may fail in case very large
entries are stored in the cache
Key: IGNITE-12510
URL: https://issues.apache.org/jira/browse/IGNITE-12510
Project: Ignite
Issue Type: Bug
Affects Versions: 2.7.6
Reporter: Ivan Rakov
In-memory page eviction (both DataPageEvictionMode#RANDOM_LRU and
DataPageEvictionMode#RANDOM_2_LRU) has limited number of attempts to choose
candidate page for data removal:
{code:java}
if (sampleSpinCnt > SAMPLE_SPIN_LIMIT) { // 5000
LT.warn(log, "Too many attempts to choose data page: " +
SAMPLE_SPIN_LIMIT);
return;
}
{code}
Large data entries are stored in several data pages which are sequentially
linked to each other. Only "head" pages are suitable for eviction, because the
whole entry is available only from "head" page (list of pages is singly linked;
there are no reverse links from tail to head).
The problem is that if we put large enough entries to evictable cache (e.g.
each entry needs more than 5000 pages to be stored), there are too few head
pages and "Too many attempts to choose data page" error is likely to show up.
We need to perform something like full scan if we failed to find a head page in
SAMPLE_SPIN_LIMIT attempts instead of just failing node with error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12509) CACHE_REBALANCE_STOPPED event raises for wrong caches in case of specified RebalanceDelay
Ivan Rakov created IGNITE-12509: --- Summary: CACHE_REBALANCE_STOPPED event raises for wrong caches in case of specified RebalanceDelay Key: IGNITE-12509 URL: https://issues.apache.org/jira/browse/IGNITE-12509 Project: Ignite Issue Type: Bug Reporter: Ivan Rakov Fix For: 2.9 Steps to reproduce: 1. Start in-memory cluster with 2 server nodes 2. Start 3 caches with different rebalance delays (e.g. 5, 10 and 15 seconds) and upload some data 3. Start localListener for EVT_CACHE_REBALANCE_STOPPED event on one of the nodes. 4. Start one more server node. 5. Wait for 5 seconds, till rebalance delay is reached. 6. EVT_CACHE_REBALANCE_STOPPED event received 3 times (1 for each cache), but in fact only 1 cache was rebalanced. The same happens for the rest of the caches. As result on rebalance finish we're getting event for each cache [CACHE_COUNT] times, instead of 1. Reproducer attached. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IGNITE-12508) GridCacheProcessor#cacheDescriptor(int) has O(N) complexity
Ivan Rakov created IGNITE-12508:
---
Summary: GridCacheProcessor#cacheDescriptor(int) has O(N)
complexity
Key: IGNITE-12508
URL: https://issues.apache.org/jira/browse/IGNITE-12508
Project: Ignite
Issue Type: Bug
Reporter: Ivan Rakov
Fix For: 2.9
See the method code:
{code}
@Nullable public DynamicCacheDescriptor cacheDescriptor(int cacheId) {
for (DynamicCacheDescriptor cacheDesc : cacheDescriptors().values()) {
CacheConfiguration ccfg = cacheDesc.cacheConfiguration();
assert ccfg != null : cacheDesc;
if (CU.cacheId(ccfg.getName()) == cacheId)
return cacheDesc;
}
return null;
}
{code}
This method is invoked in several hot paths which causes significant
performance regression when the number of caches is large, for example, logical
recovery and security check for indexing.
The method should be improved to use a hash map or similar data structure to
get a better complexity
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12507) Implement cache size metric in bytes
Ivan Rakov created IGNITE-12507: --- Summary: Implement cache size metric in bytes Key: IGNITE-12507 URL: https://issues.apache.org/jira/browse/IGNITE-12507 Project: Ignite Issue Type: Improvement Components: cache Reporter: Ivan Rakov Fix For: 2.9 There is a need to have cache size in bytes metric for pure in-memory case. When all data is in RAM, it is not obvious to find out exactly how much space is consumed by cache data on a running node as the only things that could be watched on is number of keys in partition on a specific node and memory usage metrics on the machine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Hello. Ivan is right that "baseline auto-adjust" is disabled by default if you start your node on existing PDS. But "baseline auto-adjust" is enabled by default for in-memory cluster due to in-memory nodes also have bound to baseline since 2.8 version. Also, I want to note that after this ticket(https://issues.apache.org/jira/projects/IGNITE/issues/IGNITE-12227). "baseline auto-adjust" would be disabled by default for any persistent cluster(empty and existed one) because current logic is a little confused and can lead to some problems which described in the ticket. -- Best regards, Anton Kalashnikov 27.12.2019, 17:58, "Ivan Bessonov" : > Hello, > > "baseline auto-adjust" is disabled by default if you start your node on > existing PDS. > It's enabled on new clusters only. > > Existing installations should not be affected by the update. Is that ok? > > пт, 27 дек. 2019 г. в 14:46, Maxim Muzafarov : > >> Ilya, >> >> +1 from my side. >> >> On Fri, 27 Dec 2019 at 14:36, Ilya Kasnacheev >> wrote: >> > >> > Hello! >> > >> > I have also noticed that we have baseline auto-adjust enabled by default >> in >> > 2.8 builds, and it breaks existing code in runtime: >> > https://issues.apache.org/jira/browse/IGNITE-12504 >> > >> > I propose to turn auto-adjust off by default in 2.8 release. What do you >> > think? >> > >> > Regards, >> > -- >> > Ilya Kasnacheev >> > >> > >> > пт, 27 дек. 2019 г. в 12:40, Sergei Ryzhov : >> > >> > > Hello! >> > > Task IGNITE-12470 is ready. >> > > https://issues.apache.org/jira/browse/IGNITE-12470 >> > > Please check this API. >> > > >> > > >> > > Regards, >> > > Ryzhov Sergei. >> > > >> > > чт, 26 дек. 2019 г. в 18:50, Maxim Muzafarov : >> > > >> > > > Ilya, >> > > > >> > > > >> > > > I agree with you that there is no risk and spring-data-2.2 can be >> > > > safely cherry-picked to the ignite-2.8 branch. I'm OK with it. Will >> > > > you do such merge or I should do it by myself? >> > > > >> > > > >> > > > As for the second part of your email, you are proposing to bump up a >> > > > minor dependencies version (no API changes) for the whole components >> > > > mentioned in the parent/pom.xml file, right? From a point of the >> > > > release view, it seems not a good thing since a scope test of the >> > > > release becomes too wider. I don't think we will simplify thus the >> > > > year-long release test scope, so as for me, this sounds not good but >> > > > I'd like to hear thoughts of other community members on this point. >> > > > >> > > > As an alternative, for instance, we can bump minor versions only for >> > > > those components which have security vulnerabilities. To find such >> > > > dependencies, I've run some local test with a maven >> > > > dependency-check-maven [1] an open-source dependency check tool. Here >> > > > is a brief report (only a few modules tested): >> > > > >> > > > spring-core-4.3.18.RELEASE.jar : CVE-2018-15756 [2] >> > > > h2-1.4.197.jar : CVE-2018-10054, CVE-2018-14335 (discussed also [3]) >> > > > ignite-shmem-1.0.0.jar : CVE-2017-14614 >> > > > >> > > > >> > > > [1] https://jeremylong.github.io/DependencyCheck/index.html >> > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2018-15756 >> > > > [3] https://issues.apache.org/jira/browse/IGNITE-10801 >> > > > >> > > > >> > > > >> > > > On Thu, 26 Dec 2019 at 15:52, Ilya Kasnacheev < >> ilya.kasnach...@gmail.com >> > > > >> > > > wrote: >> > > > > >> > > > > Hello! >> > > > > >> > > > > I propose to add the following ticket to the scope: >> > > > > https://issues.apache.org/jira/browse/IGNITE-12259 (3 commits, be >> > > > careful >> > > > > with release version) >> > > > > >> > > > > Adding tickets to scope surely seems crazy now, but I will provide >> the >> > > > > following considerations: >> > > > > * This is Spring Data 2.2 integration, which we currently do not >> have, >> > > > > leading to lots of confused questions on stack overflow and mailing >> > > list. >> > > > > Spring Data is important to our public image since many people may >> > > learn >> > > > > about out project by starting with Spring Data. >> > > > > >> > > > > * It has zero code impact outside of its own module (just 2 POM >> file >> > > > > touched and that's all). >> > > > > >> > > > > * The core was ready since early November but, due to gmail quirk, >> we >> > > did >> > > > > not react to it in time. >> > > > > >> > > > > WDYT? >> > > > > >> > > > > Another semi-related question. *Should we bump our dependencies' >> > > versions >> > > > > before releasing 2.8?* I talk mainly about spring and hibernate >> > > > > dependencies. We could switch them to their latest maintenance >> versions >> > > > to >> > > > > avoid shipping default links to outdated packages. >> > > > > >> > > > > I think this is one of things that are very hard to do between >> > > releases, >> > > > so >> > > > > I think this dependencies
Re: SQL dialects supported by Calcite
Denis, > Is it true that Calcite can parse MySQL, Oracle Yes, it can. > Do I need to select a dialect globally or can it be set on a per-query > level? Technically a parser, a validator, a planner, other components are created and configured for each query call (because they are stateful). So you may configure it per query, or hardcode a desired dialect, or get it from configuration - it’s up to you, but we expect parser configuration to be (more or less) static, it is a part of initial framework configuration. I think we should not allow such parameters (as a dialect) changing. Regards, Igor > 27 дек. 2019 г., в 07:47, Denis Magda написал(а): > > Igor S., Roman and others who involved in Calcite prototyping, > > Is it true that Calcite can parse MySQL, Oracle, ANSI-99 and other dialects > as suggested by this page? > https://calcite.apache.org/apidocs/org/apache/calcite/sql/validate/SqlConformanceEnum.html > > Do I need to select a dialect globally or can it be set on a per-query > level? > > - > Denis
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Hello, "baseline auto-adjust" is disabled by default if you start your node on existing PDS. It's enabled on new clusters only. Existing installations should not be affected by the update. Is that ok? пт, 27 дек. 2019 г. в 14:46, Maxim Muzafarov : > Ilya, > > +1 from my side. > > On Fri, 27 Dec 2019 at 14:36, Ilya Kasnacheev > wrote: > > > > Hello! > > > > I have also noticed that we have baseline auto-adjust enabled by default > in > > 2.8 builds, and it breaks existing code in runtime: > > https://issues.apache.org/jira/browse/IGNITE-12504 > > > > I propose to turn auto-adjust off by default in 2.8 release. What do you > > think? > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > пт, 27 дек. 2019 г. в 12:40, Sergei Ryzhov : > > > > > Hello! > > > Task IGNITE-12470 is ready. > > > https://issues.apache.org/jira/browse/IGNITE-12470 > > > Please check this API. > > > > > > > > > Regards, > > > Ryzhov Sergei. > > > > > > чт, 26 дек. 2019 г. в 18:50, Maxim Muzafarov : > > > > > > > Ilya, > > > > > > > > > > > > I agree with you that there is no risk and spring-data-2.2 can be > > > > safely cherry-picked to the ignite-2.8 branch. I'm OK with it. Will > > > > you do such merge or I should do it by myself? > > > > > > > > > > > > As for the second part of your email, you are proposing to bump up a > > > > minor dependencies version (no API changes) for the whole components > > > > mentioned in the parent/pom.xml file, right? From a point of the > > > > release view, it seems not a good thing since a scope test of the > > > > release becomes too wider. I don't think we will simplify thus the > > > > year-long release test scope, so as for me, this sounds not good but > > > > I'd like to hear thoughts of other community members on this point. > > > > > > > > As an alternative, for instance, we can bump minor versions only for > > > > those components which have security vulnerabilities. To find such > > > > dependencies, I've run some local test with a maven > > > > dependency-check-maven [1] an open-source dependency check tool. Here > > > > is a brief report (only a few modules tested): > > > > > > > > spring-core-4.3.18.RELEASE.jar : CVE-2018-15756 [2] > > > > h2-1.4.197.jar : CVE-2018-10054, CVE-2018-14335 (discussed also [3]) > > > > ignite-shmem-1.0.0.jar : CVE-2017-14614 > > > > > > > > > > > > [1] https://jeremylong.github.io/DependencyCheck/index.html > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2018-15756 > > > > [3] https://issues.apache.org/jira/browse/IGNITE-10801 > > > > > > > > > > > > > > > > On Thu, 26 Dec 2019 at 15:52, Ilya Kasnacheev < > ilya.kasnach...@gmail.com > > > > > > > > wrote: > > > > > > > > > > Hello! > > > > > > > > > > I propose to add the following ticket to the scope: > > > > > https://issues.apache.org/jira/browse/IGNITE-12259 (3 commits, be > > > > careful > > > > > with release version) > > > > > > > > > > Adding tickets to scope surely seems crazy now, but I will provide > the > > > > > following considerations: > > > > > * This is Spring Data 2.2 integration, which we currently do not > have, > > > > > leading to lots of confused questions on stack overflow and mailing > > > list. > > > > > Spring Data is important to our public image since many people may > > > learn > > > > > about out project by starting with Spring Data. > > > > > > > > > > * It has zero code impact outside of its own module (just 2 POM > file > > > > > touched and that's all). > > > > > > > > > > * The core was ready since early November but, due to gmail quirk, > we > > > did > > > > > not react to it in time. > > > > > > > > > > WDYT? > > > > > > > > > > Another semi-related question. *Should we bump our dependencies' > > > versions > > > > > before releasing 2.8?* I talk mainly about spring and hibernate > > > > > dependencies. We could switch them to their latest maintenance > versions > > > > to > > > > > avoid shipping default links to outdated packages. > > > > > > > > > > I think this is one of things that are very hard to do between > > > releases, > > > > so > > > > > I think this dependencies bumping should be a part of a formal > > > > > release/testing cycle, and then be backported to master. > > > > > > > > > > I could volunteer to do that myself, if we agree to merge these > version > > > > > upgrades to ignite-2.8 and then re-test. > > > > > > > > > > Regards, > > > > > -- > > > > > Ilya Kasnacheev > > > > > > > > > > > > > > > вт, 24 дек. 2019 г. в 13:22, Zhenya Stanilovsky > > > > > > > > >: > > > > > > > > > > > > > > > > > Igniters, i`l try to compare 2.8 release candidate vs 2.7.6, > > > > > > last sha 2.8 was build from : 9d114f3137f92aebc2562a > > > > > > i use yardstick benchmarks, 4 bare machine with: 2x Xeon X5570 > 96Gb > > > > 512GB > > > > > > SSD 2048GB HDD 10GB/s > > > > > > 1 for client (driver) and 3 for servers. > > > > > > this mappings for graphs and real yardstick tests: > > > > > > > > > > > > atomic-put: IgnitePutBenchmark > > > > > >
[jira] [Created] (IGNITE-12506) Deadlock in DistributedMetaStoragePersistentTest.testUnstableTopology
Ivan Bessonov created IGNITE-12506:
--
Summary: Deadlock in
DistributedMetaStoragePersistentTest.testUnstableTopology
Key: IGNITE-12506
URL: https://issues.apache.org/jira/browse/IGNITE-12506
Project: Ignite
Issue Type: Bug
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov
Fix For: 2.9
{code:java}
"wal-file-archiver%metastorage.DistributedMetaStoragePersistentTest4-#51609%metastorage.DistributedMetaStoragePersistentTest4%@88463"
prio=5 tid=0xf889 nid=NA waiting for monitor entry
java.lang.Thread.State: BLOCKED
waiting for
dms-writer-thread-#51614%metastorage.DistributedMetaStoragePersistentTest4%@88460
to release lock on <0x159fe> (a
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver$2.apply(FileWriteAheadLogManager.java:2042)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver$2.apply(FileWriteAheadLogManager.java:2040)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.checkFiles(FileWriteAheadLogManager.java:2538)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.access$3000(FileWriteAheadLogManager.java:157)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.allocateRemainingFiles(FileWriteAheadLogManager.java:2032)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.body(FileWriteAheadLogManager.java:1806)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at java.lang.Thread.run(Thread.java:748)
{code}
{code:java}
"dms-writer-thread-#51614%metastorage.DistributedMetaStoragePersistentTest4%@88460"
prio=5 tid=0xf88e nid=NA waiting java.lang.Thread.State: WAITING blocks
wal-file-archiver%metastorage.DistributedMetaStoragePersistentTest4-#51609%metastorage.DistributedMetaStoragePersistentTest4%@88463
at java.lang.Object.wait(Object.java:-1) at
java.lang.Object.wait(Object.java:502) at
org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.nextAbsoluteSegmentIndex(SegmentCurrentStateStorage.java:107)
at
org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentAware.nextAbsoluteSegmentIndex(SegmentAware.java:66)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.nextAbsoluteSegmentIndex(FileWriteAheadLogManager.java:1918)
- locked <0x159fe> (a
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.access$1100(FileWriteAheadLogManager.java:1687)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.pollNextFile(FileWriteAheadLogManager.java:1575)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.initNextWriteHandle(FileWriteAheadLogManager.java:1387)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.rollOver(FileWriteAheadLogManager.java:1258)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:875)
at
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.log(FileWriteAheadLogManager.java:796)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList$WriteRowHandler.addRow(AbstractFreeList.java:207)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList$WriteRowHandler.run(AbstractFreeList.java:158)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList$WriteRowHandler.run(AbstractFreeList.java:138)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:292)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:318)
at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.insertDataRow(AbstractFreeList.java:516)
at
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetastorageRowStore.addRow(MetastorageRowStore.java:72)
at
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.writeRaw(MetaStorage.java:419)
- locked <0x159ff> (a
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage)
at
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.write(MetaStorage.java:396)
at
Re: Improve logging of Data Region configuration
Hello! I have filed a ticket about this improvement, plan to start on coding. https://issues.apache.org/jira/browse/IGNITE-12505 Regards, -- Ilya Kasnacheev чт, 26 дек. 2019 г. в 12:50, Ilya Kasnacheev : > Hello! > > Okay, I will mark default region. > > We already log information about internal memory regions, so removing it > will require a lot of consensus. However, I can clearly map them as system > regions when printing. > > Regards, > -- > Ilya Kasnacheev > > > чт, 26 дек. 2019 г. в 12:10, Ivan Pavlukhin : > >> Ilya, >> >> Indeed the matters can be improved. >> >> Is not it useful to mark what region is default? Also some doubts >> about internal memory regions. It is not obvious that we should print >> an information about them for every user. If we need to have some >> determinism about offheap memory than I can think about logging >> amounts for internal needs of total ones (a sum for all regions). >> >> вт, 24 дек. 2019 г. в 15:38, Ilya Kasnacheev : >> > >> > Hello! >> > >> > It came to my attention that we output data regions' configurations >> twice >> > when starting node, but we never output list of data regions (including >> > system, etc) that were actually started. >> > >> > First we have IgniteConfiguration printed (quiet=false): >> > 2019-07-24 02:33:33.918[INFO >> ][Thread-139][o.a.i.i.IgniteKernal%GridNodeName >> > ] IgniteConfiguration [... dfltDataRegConf=DataRegionConfiguration >> [name= >> > mem_plc, maxSize=635655159808, initSize=268435456, swapPath=null, >> > pageEvictionMode=DISABLED, evictionThreshold=0.9, >> emptyPagesPoolSize=100, >> > metricsEnabled=true, metricsSubIntervalCount=5, >> metricsRateTimeInterval=1000 >> > , persistenceEnabled=true, checkpointPageBufSize=17179869184], >> storagePath=/ >> > ssd/data, checkpointFreq=3, lockWaitTime=1, checkpointThreads=4, >> > checkpointWriteOrder=SEQUENTIAL, walHistSize=2147483647, walSegments=10, >> > walSegmentSize=1073741824, walPath=/ssd/data/wal, walArchivePath=/sas/ >> > wal_archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, >> > walBuffSize=5242880, walFlushFreq=2000, walFsyncDelay=1000, >> > walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, >> fileIOFactory= >> > org.apache.ignite.internal.processors.cache.persistence.file. >> > AsyncFileIOFactory@3612c49a, metricsSubIntervalCnt=5, >> > metricsRateTimeInterval=6, walAutoArchiveAfterInactivity=-1, >> > writeThrottlingEnabled=false, walCompactionEnabled=true, >> walCompactionLevel= >> > 1], ...] >> > >> > Then we have all configured Data Regions printed per IGNITE-8803 >> > (quiet=true): >> > [11:30:36] Data Regions Configured: >> > [11:30:36] ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB, >> > persistence=false, lazyMemoryAllocation=true] >> > [11:30:36] ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB, >> > persistence=false, lazyMemoryAllocation=true] >> > >> > Then we print number of Data Regions that were initialized as per >> > IGNITE-7196, but not regions themselves (quiet=false): >> > Configured data regions initialized successfully [total=4] >> > >> > I propose to keep the first one (IgniteConfiguration), remove the second >> > one (Data Regions Configured), and promote the last one to quiet mode >> while >> > also outputting the regions themselves like this: >> > [11:30:36] Data Regions Initialized Successfully: 4 >> > [11:30:36] ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB, >> > persistence=true, lazyMemoryAllocation=true] >> > [11:30:36] ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB, >> > persistence=true, lazyMemoryAllocation=true] >> > [11:30:36] ^-- sysMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB, >> > persistence=true, lazyMemoryAllocation=false] >> > [11:30:36] ^-- volatileMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB, >> > persistence=false, lazyMemoryAllocation=true] >> > >> > (maybe it will also include information about current usage of region in >> > line with IGNITE-9305's "Metrics for local node" >> > >> > WDYT? >> > >> > Regards, >> >> >> >> -- >> Best regards, >> Ivan Pavlukhin >> >
[jira] [Created] (IGNITE-12505) Improve logging of data regions on startup
Ilya Kasnacheev created IGNITE-12505:
Summary: Improve logging of data regions on startup
Key: IGNITE-12505
URL: https://issues.apache.org/jira/browse/IGNITE-12505
Project: Ignite
Issue Type: Improvement
Reporter: Ilya Kasnacheev
Assignee: Ilya Kasnacheev
First we have IgniteConfiguration printed (quiet=false):
{quote}2019-07-24 02:33:33.918[INFO
][Thread-139][o.a.i.i.IgniteKernal%GridNodeName] IgniteConfiguration [...
dfltDataRegConf=DataRegionConfiguration [name=mem_plc, maxSize=635655159808,
initSize=268435456, swapPath=null, pageEvictionMode=DISABLED,
evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=true,
metricsSubIntervalCount=5, metricsRateTimeInterval=1000,
persistenceEnabled=true, checkpointPageBufSize=17179869184],
storagePath=/ssd/data, checkpointFreq=3, lockWaitTime=1,
checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=2147483647,
walSegments=10, walSegmentSize=1073741824, walPath=/ssd/data/wal,
walArchivePath=/sas/wal_archive, metricsEnabled=false, walMode=LOG_ONLY,
walTlbSize=131072, walBuffSize=5242880, walFlushFreq=2000, walFsyncDelay=1000,
walRecordIterBuffSize=67108864, alwaysWriteFullPages=false,
fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@3612c49a,
metricsSubIntervalCnt=5, metricsRateTimeInterval=6,
walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false,
walCompactionEnabled=true, walCompactionLevel=1], ...]{quote}
Then we have all configured Data Regions printed per IGNITE-8803 (quiet=true):
{quote} [11:30:36] Data Regions Configured:
[11:30:36] ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
persistence=false, lazyMemoryAllocation=true]
[11:30:36] ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
persistence=false, lazyMemoryAllocation=true]{quote}
Then we print number of Data Regions that were initialized as per IGNITE-7196,
but not regions themselves (quiet=false):
Configured data regions initialized successfully [total=4]
I propose to keep the first one (IgniteConfiguration), remove the second one
(Data Regions Configured), and promote the last one to quiet mode while also
outputting the regions themselves like this:
{quote} [11:30:36] Data Regions Initialized Successfully: 4
[11:30:36] ^-- plcWithMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
persistence=true, lazyMemoryAllocation=true]
[11:30:36] ^-- plcNoMetrics [initSize=256,0 MiB, maxSize=6,3 GiB,
persistence=true, lazyMemoryAllocation=true]
[11:30:36] ^-- sysMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
persistence=true, lazyMemoryAllocation=false]
[11:30:36] ^-- volatileMemPlc [initSize=40,0 MiB, maxSize=100,0 MiB,
persistence=false, lazyMemoryAllocation=true]{quote}
Also clearly mark default region and system regions as such.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Ilya, +1 from my side. On Fri, 27 Dec 2019 at 14:36, Ilya Kasnacheev wrote: > > Hello! > > I have also noticed that we have baseline auto-adjust enabled by default in > 2.8 builds, and it breaks existing code in runtime: > https://issues.apache.org/jira/browse/IGNITE-12504 > > I propose to turn auto-adjust off by default in 2.8 release. What do you > think? > > Regards, > -- > Ilya Kasnacheev > > > пт, 27 дек. 2019 г. в 12:40, Sergei Ryzhov : > > > Hello! > > Task IGNITE-12470 is ready. > > https://issues.apache.org/jira/browse/IGNITE-12470 > > Please check this API. > > > > > > Regards, > > Ryzhov Sergei. > > > > чт, 26 дек. 2019 г. в 18:50, Maxim Muzafarov : > > > > > Ilya, > > > > > > > > > I agree with you that there is no risk and spring-data-2.2 can be > > > safely cherry-picked to the ignite-2.8 branch. I'm OK with it. Will > > > you do such merge or I should do it by myself? > > > > > > > > > As for the second part of your email, you are proposing to bump up a > > > minor dependencies version (no API changes) for the whole components > > > mentioned in the parent/pom.xml file, right? From a point of the > > > release view, it seems not a good thing since a scope test of the > > > release becomes too wider. I don't think we will simplify thus the > > > year-long release test scope, so as for me, this sounds not good but > > > I'd like to hear thoughts of other community members on this point. > > > > > > As an alternative, for instance, we can bump minor versions only for > > > those components which have security vulnerabilities. To find such > > > dependencies, I've run some local test with a maven > > > dependency-check-maven [1] an open-source dependency check tool. Here > > > is a brief report (only a few modules tested): > > > > > > spring-core-4.3.18.RELEASE.jar : CVE-2018-15756 [2] > > > h2-1.4.197.jar : CVE-2018-10054, CVE-2018-14335 (discussed also [3]) > > > ignite-shmem-1.0.0.jar : CVE-2017-14614 > > > > > > > > > [1] https://jeremylong.github.io/DependencyCheck/index.html > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2018-15756 > > > [3] https://issues.apache.org/jira/browse/IGNITE-10801 > > > > > > > > > > > > On Thu, 26 Dec 2019 at 15:52, Ilya Kasnacheev > > > > > wrote: > > > > > > > > Hello! > > > > > > > > I propose to add the following ticket to the scope: > > > > https://issues.apache.org/jira/browse/IGNITE-12259 (3 commits, be > > > careful > > > > with release version) > > > > > > > > Adding tickets to scope surely seems crazy now, but I will provide the > > > > following considerations: > > > > * This is Spring Data 2.2 integration, which we currently do not have, > > > > leading to lots of confused questions on stack overflow and mailing > > list. > > > > Spring Data is important to our public image since many people may > > learn > > > > about out project by starting with Spring Data. > > > > > > > > * It has zero code impact outside of its own module (just 2 POM file > > > > touched and that's all). > > > > > > > > * The core was ready since early November but, due to gmail quirk, we > > did > > > > not react to it in time. > > > > > > > > WDYT? > > > > > > > > Another semi-related question. *Should we bump our dependencies' > > versions > > > > before releasing 2.8?* I talk mainly about spring and hibernate > > > > dependencies. We could switch them to their latest maintenance versions > > > to > > > > avoid shipping default links to outdated packages. > > > > > > > > I think this is one of things that are very hard to do between > > releases, > > > so > > > > I think this dependencies bumping should be a part of a formal > > > > release/testing cycle, and then be backported to master. > > > > > > > > I could volunteer to do that myself, if we agree to merge these version > > > > upgrades to ignite-2.8 and then re-test. > > > > > > > > Regards, > > > > -- > > > > Ilya Kasnacheev > > > > > > > > > > > > вт, 24 дек. 2019 г. в 13:22, Zhenya Stanilovsky > > > > > > >: > > > > > > > > > > > > > > Igniters, i`l try to compare 2.8 release candidate vs 2.7.6, > > > > > last sha 2.8 was build from : 9d114f3137f92aebc2562a > > > > > i use yardstick benchmarks, 4 bare machine with: 2x Xeon X5570 96Gb > > > 512GB > > > > > SSD 2048GB HDD 10GB/s > > > > > 1 for client (driver) and 3 for servers. > > > > > this mappings for graphs and real yardstick tests: > > > > > > > > > > atomic-put: IgnitePutBenchmark > > > > > sql-merge-query: IgniteSqlMergeQueryBenchmark > > > > > atomic-get: IgniteGetBenchmark > > > > > tx-get: IgniteGetTxBenchmark > > > > > tx-put: IgnitePutTxBenchmark > > > > > atomic-put-all-bs-10: IgnitePutAllBenchmark > > > > > tx-put-all-bs-10: IgnitePutAllTxBenchmark > > > > > > > > > > cacheMode — partitioned > > > > > CacheWriteSynchronizationMode.FULL_SYNC > > > > > 1 backup > > > > > > > > > > 1. wal = log_only 2. wal = none 3. persistence disabled. > > > > > Thanks Maxim for wiki page [1] > > > > > > > > > > > > > > > [1] > > > > > > > > > >
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Hello! I have also noticed that we have baseline auto-adjust enabled by default in 2.8 builds, and it breaks existing code in runtime: https://issues.apache.org/jira/browse/IGNITE-12504 I propose to turn auto-adjust off by default in 2.8 release. What do you think? Regards, -- Ilya Kasnacheev пт, 27 дек. 2019 г. в 12:40, Sergei Ryzhov : > Hello! > Task IGNITE-12470 is ready. > https://issues.apache.org/jira/browse/IGNITE-12470 > Please check this API. > > > Regards, > Ryzhov Sergei. > > чт, 26 дек. 2019 г. в 18:50, Maxim Muzafarov : > > > Ilya, > > > > > > I agree with you that there is no risk and spring-data-2.2 can be > > safely cherry-picked to the ignite-2.8 branch. I'm OK with it. Will > > you do such merge or I should do it by myself? > > > > > > As for the second part of your email, you are proposing to bump up a > > minor dependencies version (no API changes) for the whole components > > mentioned in the parent/pom.xml file, right? From a point of the > > release view, it seems not a good thing since a scope test of the > > release becomes too wider. I don't think we will simplify thus the > > year-long release test scope, so as for me, this sounds not good but > > I'd like to hear thoughts of other community members on this point. > > > > As an alternative, for instance, we can bump minor versions only for > > those components which have security vulnerabilities. To find such > > dependencies, I've run some local test with a maven > > dependency-check-maven [1] an open-source dependency check tool. Here > > is a brief report (only a few modules tested): > > > > spring-core-4.3.18.RELEASE.jar : CVE-2018-15756 [2] > > h2-1.4.197.jar : CVE-2018-10054, CVE-2018-14335 (discussed also [3]) > > ignite-shmem-1.0.0.jar : CVE-2017-14614 > > > > > > [1] https://jeremylong.github.io/DependencyCheck/index.html > > [2] https://nvd.nist.gov/vuln/detail/CVE-2018-15756 > > [3] https://issues.apache.org/jira/browse/IGNITE-10801 > > > > > > > > On Thu, 26 Dec 2019 at 15:52, Ilya Kasnacheev > > > wrote: > > > > > > Hello! > > > > > > I propose to add the following ticket to the scope: > > > https://issues.apache.org/jira/browse/IGNITE-12259 (3 commits, be > > careful > > > with release version) > > > > > > Adding tickets to scope surely seems crazy now, but I will provide the > > > following considerations: > > > * This is Spring Data 2.2 integration, which we currently do not have, > > > leading to lots of confused questions on stack overflow and mailing > list. > > > Spring Data is important to our public image since many people may > learn > > > about out project by starting with Spring Data. > > > > > > * It has zero code impact outside of its own module (just 2 POM file > > > touched and that's all). > > > > > > * The core was ready since early November but, due to gmail quirk, we > did > > > not react to it in time. > > > > > > WDYT? > > > > > > Another semi-related question. *Should we bump our dependencies' > versions > > > before releasing 2.8?* I talk mainly about spring and hibernate > > > dependencies. We could switch them to their latest maintenance versions > > to > > > avoid shipping default links to outdated packages. > > > > > > I think this is one of things that are very hard to do between > releases, > > so > > > I think this dependencies bumping should be a part of a formal > > > release/testing cycle, and then be backported to master. > > > > > > I could volunteer to do that myself, if we agree to merge these version > > > upgrades to ignite-2.8 and then re-test. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > вт, 24 дек. 2019 г. в 13:22, Zhenya Stanilovsky > > > > >: > > > > > > > > > > > Igniters, i`l try to compare 2.8 release candidate vs 2.7.6, > > > > last sha 2.8 was build from : 9d114f3137f92aebc2562a > > > > i use yardstick benchmarks, 4 bare machine with: 2x Xeon X5570 96Gb > > 512GB > > > > SSD 2048GB HDD 10GB/s > > > > 1 for client (driver) and 3 for servers. > > > > this mappings for graphs and real yardstick tests: > > > > > > > > atomic-put: IgnitePutBenchmark > > > > sql-merge-query: IgniteSqlMergeQueryBenchmark > > > > atomic-get: IgniteGetBenchmark > > > > tx-get: IgniteGetTxBenchmark > > > > tx-put: IgnitePutTxBenchmark > > > > atomic-put-all-bs-10: IgnitePutAllBenchmark > > > > tx-put-all-bs-10: IgnitePutAllTxBenchmark > > > > > > > > cacheMode — partitioned > > > > CacheWriteSynchronizationMode.FULL_SYNC > > > > 1 backup > > > > > > > > 1. wal = log_only 2. wal = none 3. persistence disabled. > > > > Thanks Maxim for wiki page [1] > > > > > > > > > > > > [1] > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks > > > > > > > > do we need some bisect or other work here ? > > > > > > > > > > > > > > > > > > >--- Forwarded message --- > > > > >From: "Maxim Muzafarov" < mmu...@apache.org > > > > > >To: dev@ignite.apache.org > > > > >Cc: > > > > >Subject: Apache Ignite 2.8 RELEASE
[jira] [Created] (IGNITE-12504) Auto-adjust breaks existing code, should be disabled by default
Ilya Kasnacheev created IGNITE-12504:
Summary: Auto-adjust breaks existing code, should be disabled by
default
Key: IGNITE-12504
URL: https://issues.apache.org/jira/browse/IGNITE-12504
Project: Ignite
Issue Type: Bug
Components: cache, persistence
Affects Versions: 2.8
Reporter: Ilya Kasnacheev
Fix For: 2.8
We have automatic baseline adjustment now. However, it is 'on' by default,
which means it breaks existing code. I see new exceptions when starting an
existing project after bumping Ignite dependency version:
{code}
Caused by:
org.apache.ignite.internal.processors.cluster.BaselineAdjustForbiddenException:
Baseline auto-adjust is enabled, please turn-off it before try to adjust
baseline manually
{code}
(Please see reproducer from attached UL discussion)
I think we should disable auto-adjust by default, let people enable it when
they see it fit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12503) Fix SSL configuration in Jetty REST: httpCfg assignment
Ilya Kasnacheev created IGNITE-12503:
Summary: Fix SSL configuration in Jetty REST: httpCfg assignment
Key: IGNITE-12503
URL: https://issues.apache.org/jira/browse/IGNITE-12503
Project: Ignite
Issue Type: Bug
Components: rest
Reporter: Ilya Kasnacheev
Assignee: Ilya Kasnacheev
{code}
{code}
should become
{code}
{code}
Or this assignment is not being made.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Visor plugin
Here are my two cents on the topic: * Obviously it is bad to have two different management console utilities, so visorcmd and control.sh should be merged into a single utility * In my personal opinion, this utility should _not_ be interactive. In the current state it is impossible to use visorcmd in any kind of automation. On the other hand, I know a few users who successfully use control.sh in scripts * Daemon nodes is definitely not the way how the utility should interact with the cluster. There were a few discussions here on the dev-list related to the daemon nodes solely and they are in the "to be removed" wishlist for 3.0 [1] * We should get rid of the scala code unless it is absolutely required (like in Spark integration, for example). Multi-language modules are hard to maintain and they set significantly higher entry barrier for new and existing contributors. We may expect the utility to be used and hopefully supported by many people, so the entry barrier here should be as low as possible. Currently, scala usage in the module adds absolutely no value to the project. Having said all that, I incline to have control.sh to be used as a baseline and move functionality from visorcmd to the new utility as needed/wanted. --AG [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist пт, 27 дек. 2019 г. в 09:40, Николай Ижиков : > > 2. Move all colde from visorcmd to controls.sh > > 3. Drop visorcm > > Hello, Zhenya, Alexey, can you, please, clarify, why you suggest dropping > of visorcmd? > What are the problems of the visorcmd that can’t be solved? > > > > 27 дек. 2019 г., в 09:26, Zhenya Stanilovsky > написал(а): > > > > > > +1 here. > > > > > >> Hi, All! > >> > >> I think that the best way is to do the following: > >> 1. Move controls.sh to separate module (this will allow us to use any > third > >> party libs for argument parsing, interactive mode, and other stuff). > >> 2. Move all colde from visorcmd to controls.sh > >> 3. Drop visorcmd > >> > >> On Thu, Dec 26, 2019 at 11:29 PM Николай Ижиков < nizhi...@apache.org > > wrote: > >> > >>> +1 to keep visor cmd and fix it’s issues. > >>> > 26 дек. 2019 г., в 19:23, Denis Magda < dma...@apache.org > > написал(а): > > Personally, I see no reason for deprecating Visor CLI and moving all > its > capabilities to the control.sh script. It's better to merge the > script's > capabilities into Visor CLI and rework the connectivity/communication > protocol of the latter. If to recall the reason for the control.sh > >>> creation > it was the Visor's daemon-mode way of interaction with the cluster > that > >>> is > cumbersome and complicated. > > - > Denis > > > On Thu, Dec 26, 2019 at 1:49 AM Ilya Kasnacheev < > >>> ilya.kasnach...@gmail.com > > wrote: > > > Hello! > > > > I think that control.sh should be extended instead of Visor CLI, it > is a > > tool which sees a lot more activity currently. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > чт, 26 дек. 2019 г. в 12:45, Ivan Pavlukhin < vololo...@gmail.com >: > > > >> Folks, > >> > >> Do not we have plans to discontinue Visor? If so, it might be better > >> to add a desired functionality to another management API? > >> > >> вт, 24 дек. 2019 г. в 08:02, Denis Magda < dma...@apache.org >: > >>> > >>> Forwarding to the dev list. How exactly would you like to expand > Visor > >> CMD? > >>> Please describe your idea and we can mover from that point. > >>> > >>> Denis > >>> > >>> On Monday, December 23, 2019, sgtech19 < rajado...@gmail.com > > wrote: > >>> > Hello team, > I would like to add a new feature to the existing > >> visor > commands. Could you give me some direction on how to achieve this > if > >> its > possible? Do we need a visor plugin ? if so,please provide any > > example > >> of > this plugin . > > Thanks > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > > >>> > >>> > >>> -- > >>> - > >>> Denis > >> > >> > >> > >> -- > >> Best regards, > >> Ivan Pavlukhin > >> > > > >>> > >>> > >> -- > >> Alexey Kuznetsov > >> > > > > > > > > > >
Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]
Hello! Task IGNITE-12470 is ready. https://issues.apache.org/jira/browse/IGNITE-12470 Please check this API. Regards, Ryzhov Sergei. чт, 26 дек. 2019 г. в 18:50, Maxim Muzafarov : > Ilya, > > > I agree with you that there is no risk and spring-data-2.2 can be > safely cherry-picked to the ignite-2.8 branch. I'm OK with it. Will > you do such merge or I should do it by myself? > > > As for the second part of your email, you are proposing to bump up a > minor dependencies version (no API changes) for the whole components > mentioned in the parent/pom.xml file, right? From a point of the > release view, it seems not a good thing since a scope test of the > release becomes too wider. I don't think we will simplify thus the > year-long release test scope, so as for me, this sounds not good but > I'd like to hear thoughts of other community members on this point. > > As an alternative, for instance, we can bump minor versions only for > those components which have security vulnerabilities. To find such > dependencies, I've run some local test with a maven > dependency-check-maven [1] an open-source dependency check tool. Here > is a brief report (only a few modules tested): > > spring-core-4.3.18.RELEASE.jar : CVE-2018-15756 [2] > h2-1.4.197.jar : CVE-2018-10054, CVE-2018-14335 (discussed also [3]) > ignite-shmem-1.0.0.jar : CVE-2017-14614 > > > [1] https://jeremylong.github.io/DependencyCheck/index.html > [2] https://nvd.nist.gov/vuln/detail/CVE-2018-15756 > [3] https://issues.apache.org/jira/browse/IGNITE-10801 > > > > On Thu, 26 Dec 2019 at 15:52, Ilya Kasnacheev > wrote: > > > > Hello! > > > > I propose to add the following ticket to the scope: > > https://issues.apache.org/jira/browse/IGNITE-12259 (3 commits, be > careful > > with release version) > > > > Adding tickets to scope surely seems crazy now, but I will provide the > > following considerations: > > * This is Spring Data 2.2 integration, which we currently do not have, > > leading to lots of confused questions on stack overflow and mailing list. > > Spring Data is important to our public image since many people may learn > > about out project by starting with Spring Data. > > > > * It has zero code impact outside of its own module (just 2 POM file > > touched and that's all). > > > > * The core was ready since early November but, due to gmail quirk, we did > > not react to it in time. > > > > WDYT? > > > > Another semi-related question. *Should we bump our dependencies' versions > > before releasing 2.8?* I talk mainly about spring and hibernate > > dependencies. We could switch them to their latest maintenance versions > to > > avoid shipping default links to outdated packages. > > > > I think this is one of things that are very hard to do between releases, > so > > I think this dependencies bumping should be a part of a formal > > release/testing cycle, and then be backported to master. > > > > I could volunteer to do that myself, if we agree to merge these version > > upgrades to ignite-2.8 and then re-test. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > вт, 24 дек. 2019 г. в 13:22, Zhenya Stanilovsky > > >: > > > > > > > > Igniters, i`l try to compare 2.8 release candidate vs 2.7.6, > > > last sha 2.8 was build from : 9d114f3137f92aebc2562a > > > i use yardstick benchmarks, 4 bare machine with: 2x Xeon X5570 96Gb > 512GB > > > SSD 2048GB HDD 10GB/s > > > 1 for client (driver) and 3 for servers. > > > this mappings for graphs and real yardstick tests: > > > > > > atomic-put: IgnitePutBenchmark > > > sql-merge-query: IgniteSqlMergeQueryBenchmark > > > atomic-get: IgniteGetBenchmark > > > tx-get: IgniteGetTxBenchmark > > > tx-put: IgnitePutTxBenchmark > > > atomic-put-all-bs-10: IgnitePutAllBenchmark > > > tx-put-all-bs-10: IgnitePutAllTxBenchmark > > > > > > cacheMode — partitioned > > > CacheWriteSynchronizationMode.FULL_SYNC > > > 1 backup > > > > > > 1. wal = log_only 2. wal = none 3. persistence disabled. > > > Thanks Maxim for wiki page [1] > > > > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8#ApacheIgnite2.8-Benchmarks > > > > > > do we need some bisect or other work here ? > > > > > > > > > > > > > > >--- Forwarded message --- > > > >From: "Maxim Muzafarov" < mmu...@apache.org > > > > >To: dev@ignite.apache.org > > > >Cc: > > > >Subject: Apache Ignite 2.8 RELEASE [Time, Scope, Manager] > > > >Date: Fri, 20 Sep 2019 14:44:31 +0300 > > > > > > > >Igniters, > > > > > > > > > > > >It's almost a year has passed since the last major Apache Ignite 2.7 > > > >has been released. We've accumulated a lot of performance improvements > > > >and a lot of new features which are waiting for their release date. > > > >Here is my list of the most interesting things from my point since the > > > >last major release: > > > > > > > >Service Grid, > > > >Monitoring, > > > >Recovery Read > > > >BLT auto-adjust, > > > >PDS compression, > > > >WAL page compression, > > > >Thin client: best
