[jira] [Updated] (GEODE-10427) fake weight when server rebuild
[ https://issues.apache.org/jira/browse/GEODE-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiuzhou liu updated GEODE-10427: Affects Version/s: 1.14.3 > fake weight when server rebuild > --- > > Key: GEODE-10427 > URL: https://issues.apache.org/jira/browse/GEODE-10427 > Project: Geode > Issue Type: Bug > Components: client/server >Affects Versions: 1.14.3, 1.14.4, 1.15.1 >Reporter: jiuzhou liu >Priority: Major > Labels: needsTriage > > The 1+1+1 design we used locked one server during the reinstallation process, > but after locking it, we found that the wrong region was calculated in its > data and passed to the other two servers, resulting in the other two servers > stopped serving. We want to know the following two questions: > how the weight value is calculated? > how does this weight transfer to another geored site? > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (GEODE-10427) fake weight when server rebuild
[ https://issues.apache.org/jira/browse/GEODE-10427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Murmann updated GEODE-10427: -- Labels: needsTriage (was: ) > fake weight when server rebuild > --- > > Key: GEODE-10427 > URL: https://issues.apache.org/jira/browse/GEODE-10427 > Project: Geode > Issue Type: Bug > Components: client/server >Affects Versions: 1.14.4, 1.15.1 >Reporter: jiuzhou liu >Priority: Major > Labels: needsTriage > > The 1+1+1 design we used locked one server during the reinstallation process, > but after locking it, we found that the wrong region was calculated in its > data and passed to the other two servers, resulting in the other two servers > stopped serving. We want to know the following two questions: > how the weight value is calculated? > how does this weight transfer to another geored site? > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (GEODE-10427) fake weight when server rebuild
jiuzhou liu created GEODE-10427: --- Summary: fake weight when server rebuild Key: GEODE-10427 URL: https://issues.apache.org/jira/browse/GEODE-10427 Project: Geode Issue Type: Bug Components: client/server Affects Versions: 1.15.1, 1.14.4 Reporter: jiuzhou liu The 1+1+1 design we used locked one server during the reinstallation process, but after locking it, we found that the wrong region was calculated in its data and passed to the other two servers, resulting in the other two servers stopped serving. We want to know the following two questions: how the weight value is calculated? how does this weight transfer to another geored site? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10323) OffHeapStorageJUnitTest testCreateOffHeapStorage fails with AssertionError: expected:<100> but was:<1048576>
[
https://issues.apache.org/jira/browse/GEODE-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen Nichols closed GEODE-10323.
> OffHeapStorageJUnitTest testCreateOffHeapStorage fails with AssertionError:
> expected:<100> but was:<1048576>
>
>
> Key: GEODE-10323
> URL: https://issues.apache.org/jira/browse/GEODE-10323
> Project: Geode
> Issue Type: Bug
> Components: offheap
>Affects Versions: 1.16.0
>Reporter: Kirk Lund
>Assignee: Alberto Gomez
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> [Please see 1st comment on this ticket below the description for
> recommendation on how to fix.]
> {{OffHeapStorageJUnitTest testCreateOffHeapStorage}} started failing
> intermittently due to commit a350ed2 which went in as
> "[GEODE-10087|https://issues.apache.org/jira/browse/GEODE-10087]: Enhance
> off-heap fragmentation visibility.
> ([#7407|https://github.com/apache/geode/pull/7407/commits/1640effc5c760afa8d9ec4c2743950bb1de0ef8f])".
> The failure stack looks like:
> {noformat}
> OffHeapStorageJUnitTest > testCreateOffHeapStorage FAILED
> java.lang.AssertionError: expected:<100> but was:<1048576>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:633)
> at
> org.apache.geode.internal.offheap.OffHeapStorageJUnitTest.testCreateOffHeapStorage(OffHeapStorageJUnitTest.java:220)
> {noformat}
> The cause is in {{MemoryAllocatorImpl}}. A new scheduled executor
> {{updateNonRealTimeStatsExecutor}} was added near the bottom of the
> constructor which immediately gets scheduled to invoke
> {{freeList::updateNonRealTimeStats}} repeatedly at the specified frequency of
> {{updateOffHeapStatsFrequencyMs}} whenever an instance of
> {{MemoryAllocatorImpl}} is created.
> {{freeList::updateNonRealTimeStats}} then updates {{setLargestFragment}} and
> {{setFreedChunks}}.
> Scheduling or starting any sort of threads from a constructor is considered a
> bad practice and should only be done via some sort of activation method such
> as {{start()}} which can be invoked from the static factory method for
> {{MemoryAllocatorImpl}}. The unit tests would then continue to construct
> instances via a constructor that does NOT invoke {{start()}}.
> {{OffHeapStorageJUnitTest testCreateOffHeapStorage}}, and possibly other
> tests, is written with the assumption that no other thread or component can
> or will be updating the {{OffHeapStats}}. Hence it is then safe for the test
> to setLargestFragment and then assert that it has the value passed in:
> {noformat}
> 219: stats.setLargestFragment(100);
> 220: assertEquals(100, stats.getLargestFragment());
> {noformat}
> Line 220 is the source of the assertion failure because
> {{updateNonRealTimeStatsExecutor}} is racing with the JUnit thread and
> changes the value of {{setLargestFragment}} immediately after the test sets
> the value to 100, but before the test invokes the assertion.
> Looking back at the [PR test
> results|https://cio.hdb.gemfire-ci.info/commits/pr/7407/junit?days=90] we can
> see that stress-new-test failed 5 times with this exact failure before being
> merged to develop:
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646140652/
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646154134/
> (x2)
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646156997/
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646170346/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen Nichols closed GEODE-10401.
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (GEODE-10411) XSS vulnerabiltiy in Pulse data browser
[ https://issues.apache.org/jira/browse/GEODE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10411. > XSS vulnerabiltiy in Pulse data browser > --- > > Key: GEODE-10411 > URL: https://issues.apache.org/jira/browse/GEODE-10411 > Project: Geode > Issue Type: Bug > Components: pulse >Affects Versions: 1.12.9, 1.12.10, 1.13.8, 1.13.9, 1.14.4, 1.14.5, 1.15.0, > 1.15.1, 1.16.0 >Reporter: Joris Melchior >Assignee: Joris Melchior >Priority: Major > Labels: pull-request-available > Fix For: 1.12.10, 1.13.9, 1.14.5, 1.15.1, 1.16.0 > > > # Description: > Stored XSS via data injection into Geode database, the injected > payload eventually gets executed on Pulse web application when the > admin querying data from Geode. > # PoC: > Step 1: With Geode up and running, run gfsh command to get into > interactive mode: > shell$ gfsh > Step 2: In gfsh console, execute the following command to insert a > data entry into regionA (assume that regionA is created before). Note > that the value of this data entry contains JavaScript code: > gfsh> put --region=regionA --key="test" --value="alert(1)" > Step 3: Open browser to query editor of Pulse web application at > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2F192.168.93.153%3A7070%2Fpulse%2FdataBrowser.htmldata=05%7C01%7Cbakera%40vmware.com%7Cc06e6de8d92c4519303708da54fa7d03%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637915732081233095%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=ykaOkxe1hlaE7xl8XQNgBQz2%2Ful1QPxrUChoBkuaeyY%3Dreserved=0 > (assume that already > logged in as admin), execute the following query: > SELECT * FROM /regionA > Step 4: Data from regionA will be retrieved, the XSS payload > eventually get executed > # Why this is an issue? > Developer maybe saves user-controlled data to Geode database, users > maybe submit data via an arbitrary client application (for example, a > web application), the use of gfsh console just simplifies the PoC. > # IMPACT: > Exploiting this XSS vulnerability, an attacker can steal the admin's > session cookie, therefore take over the admin account. > # CVSS: 7.6 HIGH > (https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.first.org%2Fcvss%2Fcalculator%2F3.0%23CVSS%3A3.0%2FAV%3AN%2FAC%3AL%2FPR%3AN%2FUI%3AR%2FS%3AU%2FC%3AH%2FI%3AL%2FA%3ALdata=05%7C01%7Cbakera%40vmware.com%7Cc06e6de8d92c4519303708da54fa7d03%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637915732081233095%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=W5dDA8kMdT1IVeUVX6mhWHhZ2HnAZbXErEB%2F0Tjs5hg%3Dreserved=0 > ) > (re-calculate if not correct) > # Fix: > The Pulse web application must URL encode data retrieved from Geode database. > # Credit: > The issue is found by Nguyen Thai Hung (@nth347), Viettel Cyber Security. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10281) Internal conflict replicated region resolution causes data inconsistency between two sites
[
https://issues.apache.org/jira/browse/GEODE-10281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen Nichols closed GEODE-10281.
> Internal conflict replicated region resolution causes data inconsistency
> between two sites
> --
>
> Key: GEODE-10281
> URL: https://issues.apache.org/jira/browse/GEODE-10281
> Project: Geode
> Issue Type: Bug
> Components: wan
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}It seems that the following PR
> {color}[{color:#4a6ee0}[https://github.com/apache/geode/pull/3044]{color}]{color:#0e101a}
> has not taken into consideration that event with a flag
> isConcurrencyConflict=true could delete the valid event from the queue due to
> conflation.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (GEODE-10346) Correct batch-time-interval description in documentation
[ https://issues.apache.org/jira/browse/GEODE-10346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10346. > Correct batch-time-interval description in documentation > > > Key: GEODE-10346 > URL: https://issues.apache.org/jira/browse/GEODE-10346 > Project: Geode > Issue Type: Bug > Components: docs, wan >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > The description of the batch-time-interval parameter for the Gateway Sender > in the Geode documentation states the following: > "Maximum amount of time, in ms, that can elapse before a batch is delivered." > Nevertheless, that is not completely true. > The number of ms that can elapse before a batch is delivered could be longer > than what is configured in batch-time-interval in the case that the batch > being created has not yet reached the value of max-batch-size and there are > still events in the gateway sender's queue to be added to the batch. If that > is the case, new events will keep being added to the batch without taking > into account the value for batch-time-interval. > The batch-time-interval parameter is only used when the batch being filled > has not reached the max-batch-size but there are no events in the queue. In > that case, in order not to delay the delivery of the batch until there are > events in the queue, this value is used to determine if the batch must be > sent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10291) Support Jammy (ubuntu 22.04)
[ https://issues.apache.org/jira/browse/GEODE-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10291. > Support Jammy (ubuntu 22.04) > > > Key: GEODE-10291 > URL: https://issues.apache.org/jira/browse/GEODE-10291 > Project: Geode > Issue Type: Task > Components: native client >Reporter: Michael Oleske >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > Native client should compile on the latest (ubuntu 22.04) operating system. > Consider including a new pipeline as part of this -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10371) C++ Native client: Improve dispersion on connections expiration
[ https://issues.apache.org/jira/browse/GEODE-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10371. > C++ Native client: Improve dispersion on connections expiration > --- > > Key: GEODE-10371 > URL: https://issues.apache.org/jira/browse/GEODE-10371 > Project: Geode > Issue Type: Improvement > Components: native client >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > The dispersion on connections expirations in the C++ native client works in > such a way that it adds a dispersion (variance) between -9% and 9% over the > time for a connection to expire due to load-conditioning so that, in the > event of having many connections being created at the same, they do not > expire at the right exact time. > Nevertheless, the current implementation has two problems: > - The randomness of the variance depends on the current time in seconds. As a > result, for connections created in the same second, the variance will be the > same and, therefore, the expiration time too. > - The randomness is created using the C standard's library "rand()" function > which is considered not secure. > It is recommended to change the library used to generate the random variance > to a secure one and also to make sure that for the time in seconds it does > not return the same variance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-9484) Data inconsistency in replicated region with 3 or more servers, and one server is down
[ https://issues.apache.org/jira/browse/GEODE-9484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-9484. --- > Data inconsistency in replicated region with 3 or more servers, and one > server is down > --- > > Key: GEODE-9484 > URL: https://issues.apache.org/jira/browse/GEODE-9484 > Project: Geode > Issue Type: Improvement > Components: client/server, regions >Affects Versions: 1.13.0 >Reporter: Mario Ivanac >Assignee: Mario Ivanac >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > We have configured replicated region with 3 or more servers, and client is > configured with read timeout set to value same or smaller than member timeout. > In case while client is putting data in region, one of replicated servers is > shutdown, it is observed that we have data inconsistency. > > We see that data part of data is written in server connected with client, but > in remaining replicated servers it is missing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10420) Handle WAN event when interrupted
[ https://issues.apache.org/jira/browse/GEODE-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10420. > Handle WAN event when interrupted > - > > Key: GEODE-10420 > URL: https://issues.apache.org/jira/browse/GEODE-10420 > Project: Geode > Issue Type: Bug > Components: wan >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: needsTriage, pull-request-available > Fix For: 1.15.1, 1.16.0 > > > It is possible that an event of which a gateway sender is to be notified is > lost if during the process the thread is interrupted. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10352) Update Dockerfile to use Ruby >= 2.6 in the tool to preview Geode documentation
[ https://issues.apache.org/jira/browse/GEODE-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10352. > Update Dockerfile to use Ruby >= 2.6 in the tool to preview Geode > documentation > --- > > Key: GEODE-10352 > URL: https://issues.apache.org/jira/browse/GEODE-10352 > Project: Geode > Issue Type: Bug > Components: docs >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > The script to preview the documentation of Geode (./preview-user-guide.sh) is > not working anymore. > The following error appears while running it (unless you have a previously > downloaded geodedocs/temp docker image in your local docker repo): > ERROR: Error installing elasticsearch: > The last version of faraday (>= 0) to support your Ruby & RubyGems was > 1.10.0. Try installing it with `gem install faraday -v 1.10.0` and then > running the current command again > faraday requires Ruby version >= 2.6. The current ruby version is > 2.5.9.229. > That error prevents the preview script to work. > It is needed to update the docker image used to preview the documentation to > use a Ruby version >= 2.6 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10417) Fix NullPointerException when getting events from the gw sender queue to complete transactions
[ https://issues.apache.org/jira/browse/GEODE-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10417. > Fix NullPointerException when getting events from the gw sender queue to > complete transactions > -- > > Key: GEODE-10417 > URL: https://issues.apache.org/jira/browse/GEODE-10417 > Project: Geode > Issue Type: Bug > Components: wan >Affects Versions: 1.13.8, 1.14.4, 1.15.0 >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: needsTriage, pull-request-available > Fix For: 1.15.1, 1.16.0 > > > When the WAN group-transaction-events feature is enabled in a parallel > gateway sender, it is possible to get a NullPointerException when retrieving > events from the queue to complete a transaction if the event in the queue is > null. > If this situation is reached then the gateway sender dispatcher will not > dispatch queue events anymore and therefore the WAN replication will not > progress. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10422) Add doc for using parallel recovery disk store
[ https://issues.apache.org/jira/browse/GEODE-10422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10422. > Add doc for using parallel recovery disk store > -- > > Key: GEODE-10422 > URL: https://issues.apache.org/jira/browse/GEODE-10422 > Project: Geode > Issue Type: Improvement > Components: docs >Affects Versions: 1.14.0, 1.15.0 >Reporter: Mario Kevo >Assignee: Mario Kevo >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > There are missing part for parallel recovery of disk stores. > It isn't specified that in case the user use the same disk store for the pdx > and the region, disk store recovery will go in the sequential mode. So in > case you want to use parallel disk store recovery it should be different disk > stores for the pdx and the disk store. Otherwise, it will go sequentially, > despite of default behavior to run in the parallel mode. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10415) CVEs detected in latest geode
[
https://issues.apache.org/jira/browse/GEODE-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen Nichols closed GEODE-10415.
> CVEs detected in latest geode
> -
>
> Key: GEODE-10415
> URL: https://issues.apache.org/jira/browse/GEODE-10415
> Project: Geode
> Issue Type: Bug
> Components: build
>Affects Versions: 1.15.0
>Reporter: Shruti
>Assignee: Mario Kevo
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> We are detecting the following CVEs with geode
> High or critical vulnerabilities: 21
> The spring-core is likely Not Affected. But we would like to know about the
> rest of these listed CVEs. Any info is appreciated
> {{ }}
> {{NAME INSTALLED FIXED-IN TYPE
> VULNERABILITY SEVERITY}}
> {{jetty-security 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-server 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-servlet 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-util 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-util-ajax 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-webapp 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-xml 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jgroups 3.6.14.Final 4.0.0
> java-archive GHSA-rc7h-x6cq-988q Critical}}
> {{shiro-cache 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-config-core 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-config-ogdl 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-core 1.9.0 1.9.1
> java-archive GHSA-4cf5-xmhp-3xj7 Critical}}
> {{shiro-core 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-crypto-cipher 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-crypto-core 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-crypto-hash 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-event 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-lang 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{spring-core 5.3.20
> java-archive CVE-2016-127 Critical}}
> {{jetty-http 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-io 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (GEODE-10403) Distributed deadlock when stopping gateway sender
[
https://issues.apache.org/jira/browse/GEODE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen Nichols closed GEODE-10403.
> Distributed deadlock when stopping gateway sender
> -
>
> Key: GEODE-10403
> URL: https://issues.apache.org/jira/browse/GEODE-10403
> Project: Geode
> Issue Type: Bug
> Components: wan
>Affects Versions: 1.12.9, 1.13.8, 1.14.4, 1.15.0
>Reporter: Alberto Gomez
>Assignee: Alberto Gomez
>Priority: Major
> Labels: needsTriage, pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> A distributed deadlock has been found during some tests of a Geode system
> with WAN replication when stopping the gateway sender while sending a fair
> amount of operations to the servers.
> The distributed deadlock manifests in the gateway sender stop command hanging
> forever and by all normal Geode operations from clients (gets, puts,...) not
> being responded.
> The situation is provoked by the Gateway sender stop command that first takes
> the lifecycle lock and then, at a given point, tries to retrieve the size of
> the gateway sender. This operation, that requires communication with the
> other peers never finishes, probably because the response from one of the
> peers is never received.
> Another thread is blocked when trying to acquire the lifecycle lock in
> AbstractGatewaySender.distribute().
> Finally many threads handling Geode operations (get, put...) get blocked in
> the DistributedCacheOperation._distribute() call waiting for a response from
> another peer.
> Thread dump section from blocked gateway sender stop command in call to get
> size of queue:
> {{"ConcurrentParallelGatewaySenderEventProcessor Stopper Thread1" #1316
> daemon prio=10 os_prio=0 cpu=45.55ms elapsed=4152.76s tid=0x7f92bc1c2000
> nid=0x2154 waiting on condition [0x7f9179cd2000]}}
> {{ java.lang.Thread.State: TIMED_WAITING (parking)}}
> {{ at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)}}
> {{ - parking to wait for <0x00031ca2be50> (a
> java.util.concurrent.CountDownLatch$Sync)}}
> {{ at
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.11/LockSupport.java:234)}}
> {{ at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(java.base@11.0.11/AbstractQueuedSynchronizer.java:1079)}}
> {{ at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(java.base@11.0.11/AbstractQueuedSynchronizer.java:1369)}}
> {{ at
> java.util.concurrent.CountDownLatch.await(java.base@11.0.11/CountDownLatch.java:278)}}
> {{ at
> org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)}}
> {{ at
> org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:731)}}
> {{ at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:802)}}
> {{ at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:779)}}
> {{ at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:865)}}
> {{ at
> org.apache.geode.internal.cache.partitioned.SizeMessage$SizeResponse.waitBucketSizes(SizeMessage.java:344)}}
> {{ at
> org.apache.geode.internal.cache.PartitionedRegion.getSizeRemotely(PartitionedRegion.java:6758)}}
> {{ at
> org.apache.geode.internal.cache.PartitionedRegion.entryCount(PartitionedRegion.java:6709)}}
> {{ at
> org.apache.geode.internal.cache.PartitionedRegion.entryCount(PartitionedRegion.java:6691)}}
> {{ at
> org.apache.geode.internal.cache.PartitionedRegion.getRegionSize(PartitionedRegion.java:6663)}}
> {{ at
> org.apache.geode.internal.cache.LocalRegionDataView.entryCount(LocalRegionDataView.java:99)}}
> {{ at
> org.apache.geode.internal.cache.LocalRegion.entryCount(LocalRegion.java:2078)}}
> {{ at
> org.apache.geode.internal.cache.LocalRegion.size(LocalRegion.java:8301)}}
> {{ at
> org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.size(ParallelGatewaySenderQueue.java:1670)}}
> {{ at
> org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.closeProcessor(AbstractGatewaySenderEventProcessor.java:1259)}}
> {{ at
> org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.stopProcessing(AbstractGatewaySenderEventProcessor.java:1247)}}
> {{ at
> org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor$SenderStopperCallable.call(AbstractGatewaySenderEventProcessor.java:1399)}}
> {{ at
>
[jira] [Closed] (GEODE-10155) ServerConnection thread hangs when client function execution times-out
[ https://issues.apache.org/jira/browse/GEODE-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10155. > ServerConnection thread hangs when client function execution times-out > -- > > Key: GEODE-10155 > URL: https://issues.apache.org/jira/browse/GEODE-10155 > Project: Geode > Issue Type: Bug > Components: core, functions >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > If a Geode client executes a non-HA server function (isHA=false) with a > timeout > and > the function is handled in the Geode server by a "Function Execution > Processor" thread (for example by calling the function with onRegion() on a > partitioned region without a filter) > and > the function times-out before the server has answered back with all the > results > then > the ServerConnection thread that originally started to handle the function > execution will be stuck forever. > > If this happens several times and the max-threads parameters is set to a > value greater than 0, the server will eventually run out of ServerConnection > threads and will not be able to process more client requests. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (GEODE-10404) Fix compilation for Java 11
[
https://issues.apache.org/jira/browse/GEODE-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen Nichols closed GEODE-10404.
> Fix compilation for Java 11
> ---
>
> Key: GEODE-10404
> URL: https://issues.apache.org/jira/browse/GEODE-10404
> Project: Geode
> Issue Type: Bug
> Components: native client
>Reporter: Mario Salazar de Torres
>Assignee: Mario Salazar de Torres
>Priority: Major
> Labels: needsTriage, pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> It seems that after merging
> [#973|https://github.com/apache/geode-native/pull/973], compilation with
> Geode Native Docker build image for version 1.15.0 is broken. This is the
> compilation error:
> {noformat}
> tests/javaobject/ComparePdxInstanceFunction.java:42: error: unmappable
> character (0xAC) for encoding US-ASCII
> pdxInstanceFactory.writeString("utfHugeField", longString + "???");
> {noformat}
> As it seems, the latest docker image is using Java 11, which handles UTF-8
> strings in a different way to Java 8, so that's why compilation is working
> with packer images and not with Docker build image.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (GEODE-10412) Destroy region command doesn't clear the region related expired tombstones
[
https://issues.apache.org/jira/browse/GEODE-10412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen Nichols closed GEODE-10412.
> Destroy region command doesn't clear the region related expired tombstones
> --
>
> Key: GEODE-10412
> URL: https://issues.apache.org/jira/browse/GEODE-10412
> Project: Geode
> Issue Type: Bug
> Components: expiration
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> Tombstones in geode are kept on two maps: expiredTombstones and tombstones
> (non-expired ones). When a region is destroyed, it must clear all the related
> expired and non-expired tombstones from memory. Due to the below code bug,
> expired tombstones aren't cleared when non-expired tombstones are available
> during the region destruction:
> {code:java}
> private boolean removeIf(Predicate predicate) {
> return removeUnexpiredIf(predicate) || removeExpiredIf(predicate);
> }
> {code}
> Because of the above, non-expired tombstones are never removed from memory,
> preventing other tombstones from being cleared. Since other tombstones never
> expire, the compaction is not done, and therefore the disk is filled, causing
> the issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (GEODE-10348) Correct documentation about conflation
[ https://issues.apache.org/jira/browse/GEODE-10348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen Nichols closed GEODE-10348. > Correct documentation about conflation > -- > > Key: GEODE-10348 > URL: https://issues.apache.org/jira/browse/GEODE-10348 > Project: Geode > Issue Type: Bug > Components: docs, wan >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > The Geode documentation states on conflation: > "When an update is added to a queue that has conflation enabled, if there is > already an update message in the queue for the entry key, then the existing > message assumes the value of the new update and the new update is dropped, as > shown here for key A." > Nevertheless, that is not correct. The actual behavior is the following: > "When an update is added to a queue that has conflation enabled, if there is > already an update message in the queue for the entry key, then the existing > message is removed and the new update is added to the end of the queue, as > shown here for key A." -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615335#comment-17615335
]
ASF GitHub Bot commented on GEODE-10401:
onichols-pivotal opened a new pull request, #986:
URL: https://github.com/apache/geode-native/pull/986
Native client hardcodes Geode version to test with in several places. Update
those variables to latest-and-greatest apache-geode 1.15.1
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10155) ServerConnection thread hangs when client function execution times-out
[ https://issues.apache.org/jira/browse/GEODE-10155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615315#comment-17615315 ] ASF subversion and git services commented on GEODE-10155: - Commit 0bd51e8d5db2eb0c0eb77bae0d8f2d312ae200cf in geode's branch refs/heads/master from Alberto Gomez [ https://gitbox.apache.org/repos/asf?p=geode.git;h=0bd51e8d5d ] GEODE-10155: Avoid threads hanging when function execution times-out (#7493) * GEODE-10155: Avoid threads hanging when function execution times-out * GEODE-10155: Updated after review * GEODE-10155: More changes after review * GEODE-10155: Changes after more reviews * GEODE-10155: Some more changes after review * GEODE-10155: More changes after review * GEODE-10155: More clean-up after review > ServerConnection thread hangs when client function execution times-out > -- > > Key: GEODE-10155 > URL: https://issues.apache.org/jira/browse/GEODE-10155 > Project: Geode > Issue Type: Bug > Components: core, functions >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > If a Geode client executes a non-HA server function (isHA=false) with a > timeout > and > the function is handled in the Geode server by a "Function Execution > Processor" thread (for example by calling the function with onRegion() on a > partitioned region without a filter) > and > the function times-out before the server has answered back with all the > results > then > the ServerConnection thread that originally started to handle the function > execution will be stuck forever. > > If this happens several times and the max-threads parameters is set to a > value greater than 0, the server will eventually run out of ServerConnection > threads and will not be able to process more client requests. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615325#comment-17615325
]
ASF subversion and git services commented on GEODE-10401:
-
Commit d1910cf1ee18aae749da3553b3537d74b7ef5cae in geode's branch
refs/heads/master from Jakov Varenina
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=d1910cf1ee ]
GEODE-10401: Configurable .drf recovery HashMap overflow threshold (#7828)
Configurable with the jvm parameter:
gemfire.disk.drfHashMapOverflowThreshold
Default value: 805306368
When configured threshold value is reached, then server will overflow to
the new hashmap during the recovery of .drf files. Warning: If you set
threshold parameter over 805306368, then uneeded delay will happen due
to bug in fastutil dependency.
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10089) release 1.15.0
[ https://issues.apache.org/jira/browse/GEODE-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615328#comment-17615328 ] ASF subversion and git services commented on GEODE-10089: - Commit c819951364174f78c70868dd97a4a8007d7577b6 in geode-benchmarks's branch refs/heads/master from Owen Nichols [ https://gitbox.apache.org/repos/asf?p=geode-benchmarks.git;h=c8199513 ] GEODE-10089: Bump version to 1.15.1 As part of the Geode Release Process, the geode-benchmarks build number must be rolled forward as work begins on the next release > release 1.15.0 > -- > > Key: GEODE-10089 > URL: https://issues.apache.org/jira/browse/GEODE-10089 > Project: Geode > Issue Type: Task > Components: release >Reporter: Owen Nichols >Assignee: Owen Nichols >Priority: Major > Labels: pull-request-available > > As per [Jan 25 dev list > discussion|https://lists.apache.org/thread/s9mpd207h40crcr76fpdfmohdchgdqog], > support/1.15 was cut with the intention of stabilizing and releasing a new > Geode minor. > UPDATE: As per [Mar 16 dev list > discussion|https://lists.apache.org/thread/twlwzcvmx2kqw15whpmxkh2h8bmrok21], > support/1.15 was retracted. Work will continue on develop with an eye toward > re-cutting support/1.15 in ~June > UPDATE: support/1.15 was re-cut on May 9 with an eye toward shipping in mid > to late June (2022). > Release status information is also updated in the Geode [Release > Schedule|https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615330#comment-17615330
]
ASF subversion and git services commented on GEODE-10401:
-
Commit 438131d05ff9af99436181f23f1b607b63f04573 in geode-benchmarks's branch
refs/heads/develop from Owen Nichols
[ https://gitbox.apache.org/repos/asf?p=geode-benchmarks.git;h=438131d0 ]
GEODE-10401: Update benchmark baseline
Updates the default benchmark baseline on develop to 1.15.1
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615331#comment-17615331
]
ASF subversion and git services commented on GEODE-10401:
-
Commit 29bf9c797984a32e29b2c026b2fee48b2622fccf in geode's branch
refs/heads/support/1.15 from Owen Nichols
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=29bf9c7979 ]
GEODE-10401: Bump version to 1.15.2
As part of the Geode Release Process, the build number must
be rolled forward so work can begin on the next release
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10089) release 1.15.0
[ https://issues.apache.org/jira/browse/GEODE-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615303#comment-17615303 ] ASF subversion and git services commented on GEODE-10089: - Commit 7ceaa3bfffc382fbab076e02b8f1f876b7723af2 in geode's branch refs/heads/master from Owen Nichols [ https://gitbox.apache.org/repos/asf?p=geode.git;h=7ceaa3bfff ] GEODE-10089: Add 1.15.0 as old version Add 1.15.0 in old versions and set as Benchmarks baseline on support/1.15 to enable rolling upgrade tests from 1.15.0 > release 1.15.0 > -- > > Key: GEODE-10089 > URL: https://issues.apache.org/jira/browse/GEODE-10089 > Project: Geode > Issue Type: Task > Components: release >Reporter: Owen Nichols >Assignee: Owen Nichols >Priority: Major > Labels: pull-request-available > > As per [Jan 25 dev list > discussion|https://lists.apache.org/thread/s9mpd207h40crcr76fpdfmohdchgdqog], > support/1.15 was cut with the intention of stabilizing and releasing a new > Geode minor. > UPDATE: As per [Mar 16 dev list > discussion|https://lists.apache.org/thread/twlwzcvmx2kqw15whpmxkh2h8bmrok21], > support/1.15 was retracted. Work will continue on develop with an eye toward > re-cutting support/1.15 in ~June > UPDATE: support/1.15 was re-cut on May 9 with an eye toward shipping in mid > to late June (2022). > Release status information is also updated in the Geode [Release > Schedule|https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10089) release 1.15.0
[ https://issues.apache.org/jira/browse/GEODE-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615326#comment-17615326 ] ASF subversion and git services commented on GEODE-10089: - Commit edc8a042c8be8f9951a86ad7e5b21b1cb15e840c in geode-native's branch refs/heads/master from Owen Nichols [ https://gitbox.apache.org/repos/asf?p=geode-native.git;h=edc8a042c ] GEODE-10089: Update Dockerfile and vars Native client hardcodes Geode version to test with in several places. Update native Dockerfile and other variables to apache-geode 1.15.0 > release 1.15.0 > -- > > Key: GEODE-10089 > URL: https://issues.apache.org/jira/browse/GEODE-10089 > Project: Geode > Issue Type: Task > Components: release >Reporter: Owen Nichols >Assignee: Owen Nichols >Priority: Major > Labels: pull-request-available > > As per [Jan 25 dev list > discussion|https://lists.apache.org/thread/s9mpd207h40crcr76fpdfmohdchgdqog], > support/1.15 was cut with the intention of stabilizing and releasing a new > Geode minor. > UPDATE: As per [Mar 16 dev list > discussion|https://lists.apache.org/thread/twlwzcvmx2kqw15whpmxkh2h8bmrok21], > support/1.15 was retracted. Work will continue on develop with an eye toward > re-cutting support/1.15 in ~June > UPDATE: support/1.15 was re-cut on May 9 with an eye toward shipping in mid > to late June (2022). > Release status information is also updated in the Geode [Release > Schedule|https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10323) OffHeapStorageJUnitTest testCreateOffHeapStorage fails with AssertionError: expected:<100> but was:<1048576>
[
https://issues.apache.org/jira/browse/GEODE-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615317#comment-17615317
]
ASF subversion and git services commented on GEODE-10323:
-
Commit e72da0d984a89203a3a3d5746d509d2c2c1409c1 in geode's branch
refs/heads/master from Alberto Gomez
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=e72da0d984 ]
GEODE-10323: Add small changes after review (#7819)
> OffHeapStorageJUnitTest testCreateOffHeapStorage fails with AssertionError:
> expected:<100> but was:<1048576>
>
>
> Key: GEODE-10323
> URL: https://issues.apache.org/jira/browse/GEODE-10323
> Project: Geode
> Issue Type: Bug
> Components: offheap
>Affects Versions: 1.16.0
>Reporter: Kirk Lund
>Assignee: Alberto Gomez
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> [Please see 1st comment on this ticket below the description for
> recommendation on how to fix.]
> {{OffHeapStorageJUnitTest testCreateOffHeapStorage}} started failing
> intermittently due to commit a350ed2 which went in as
> "[GEODE-10087|https://issues.apache.org/jira/browse/GEODE-10087]: Enhance
> off-heap fragmentation visibility.
> ([#7407|https://github.com/apache/geode/pull/7407/commits/1640effc5c760afa8d9ec4c2743950bb1de0ef8f])".
> The failure stack looks like:
> {noformat}
> OffHeapStorageJUnitTest > testCreateOffHeapStorage FAILED
> java.lang.AssertionError: expected:<100> but was:<1048576>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:633)
> at
> org.apache.geode.internal.offheap.OffHeapStorageJUnitTest.testCreateOffHeapStorage(OffHeapStorageJUnitTest.java:220)
> {noformat}
> The cause is in {{MemoryAllocatorImpl}}. A new scheduled executor
> {{updateNonRealTimeStatsExecutor}} was added near the bottom of the
> constructor which immediately gets scheduled to invoke
> {{freeList::updateNonRealTimeStats}} repeatedly at the specified frequency of
> {{updateOffHeapStatsFrequencyMs}} whenever an instance of
> {{MemoryAllocatorImpl}} is created.
> {{freeList::updateNonRealTimeStats}} then updates {{setLargestFragment}} and
> {{setFreedChunks}}.
> Scheduling or starting any sort of threads from a constructor is considered a
> bad practice and should only be done via some sort of activation method such
> as {{start()}} which can be invoked from the static factory method for
> {{MemoryAllocatorImpl}}. The unit tests would then continue to construct
> instances via a constructor that does NOT invoke {{start()}}.
> {{OffHeapStorageJUnitTest testCreateOffHeapStorage}}, and possibly other
> tests, is written with the assumption that no other thread or component can
> or will be updating the {{OffHeapStats}}. Hence it is then safe for the test
> to setLargestFragment and then assert that it has the value passed in:
> {noformat}
> 219: stats.setLargestFragment(100);
> 220: assertEquals(100, stats.getLargestFragment());
> {noformat}
> Line 220 is the source of the assertion failure because
> {{updateNonRealTimeStatsExecutor}} is racing with the JUnit thread and
> changes the value of {{setLargestFragment}} immediately after the test sets
> the value to 100, but before the test invokes the assertion.
> Looking back at the [PR test
> results|https://cio.hdb.gemfire-ci.info/commits/pr/7407/junit?days=90] we can
> see that stress-new-test failed 5 times with this exact failure before being
> merged to develop:
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646140652/
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646154134/
> (x2)
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646156997/
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646170346/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10323) OffHeapStorageJUnitTest testCreateOffHeapStorage fails with AssertionError: expected:<100> but was:<1048576>
[
https://issues.apache.org/jira/browse/GEODE-10323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615316#comment-17615316
]
ASF subversion and git services commented on GEODE-10323:
-
Commit 73aa4b3f72a5f517c9a281d476513a2886743299 in geode's branch
refs/heads/master from Alberto Gomez
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=73aa4b3f72 ]
GEODE-10323: Remove schedule threads in MemoryAllocatorImpl constructor (#7715)
* GEODE-10323: Remove schedule threads in MemoryAllocatorImpl
The scheduled executor used in MemoryAllocatorImpl
was scheduled in the constructor. This provoked
intermittent failures in OffHeapStorageJUnitTest testCreateOffHeapStorage
test cases due to a race condition.
The scheduling has been moved to a new method (start())
in the MemoryAllocatorImpl class that is in turn
invoked in the create() static method.
* GEODE-10323: Extract update stats code to new class
> OffHeapStorageJUnitTest testCreateOffHeapStorage fails with AssertionError:
> expected:<100> but was:<1048576>
>
>
> Key: GEODE-10323
> URL: https://issues.apache.org/jira/browse/GEODE-10323
> Project: Geode
> Issue Type: Bug
> Components: offheap
>Affects Versions: 1.16.0
>Reporter: Kirk Lund
>Assignee: Alberto Gomez
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> [Please see 1st comment on this ticket below the description for
> recommendation on how to fix.]
> {{OffHeapStorageJUnitTest testCreateOffHeapStorage}} started failing
> intermittently due to commit a350ed2 which went in as
> "[GEODE-10087|https://issues.apache.org/jira/browse/GEODE-10087]: Enhance
> off-heap fragmentation visibility.
> ([#7407|https://github.com/apache/geode/pull/7407/commits/1640effc5c760afa8d9ec4c2743950bb1de0ef8f])".
> The failure stack looks like:
> {noformat}
> OffHeapStorageJUnitTest > testCreateOffHeapStorage FAILED
> java.lang.AssertionError: expected:<100> but was:<1048576>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:633)
> at
> org.apache.geode.internal.offheap.OffHeapStorageJUnitTest.testCreateOffHeapStorage(OffHeapStorageJUnitTest.java:220)
> {noformat}
> The cause is in {{MemoryAllocatorImpl}}. A new scheduled executor
> {{updateNonRealTimeStatsExecutor}} was added near the bottom of the
> constructor which immediately gets scheduled to invoke
> {{freeList::updateNonRealTimeStats}} repeatedly at the specified frequency of
> {{updateOffHeapStatsFrequencyMs}} whenever an instance of
> {{MemoryAllocatorImpl}} is created.
> {{freeList::updateNonRealTimeStats}} then updates {{setLargestFragment}} and
> {{setFreedChunks}}.
> Scheduling or starting any sort of threads from a constructor is considered a
> bad practice and should only be done via some sort of activation method such
> as {{start()}} which can be invoked from the static factory method for
> {{MemoryAllocatorImpl}}. The unit tests would then continue to construct
> instances via a constructor that does NOT invoke {{start()}}.
> {{OffHeapStorageJUnitTest testCreateOffHeapStorage}}, and possibly other
> tests, is written with the assumption that no other thread or component can
> or will be updating the {{OffHeapStats}}. Hence it is then safe for the test
> to setLargestFragment and then assert that it has the value passed in:
> {noformat}
> 219: stats.setLargestFragment(100);
> 220: assertEquals(100, stats.getLargestFragment());
> {noformat}
> Line 220 is the source of the assertion failure because
> {{updateNonRealTimeStatsExecutor}} is racing with the JUnit thread and
> changes the value of {{setLargestFragment}} immediately after the test sets
> the value to 100, but before the test invokes the assertion.
> Looking back at the [PR test
> results|https://cio.hdb.gemfire-ci.info/commits/pr/7407/junit?days=90] we can
> see that stress-new-test failed 5 times with this exact failure before being
> merged to develop:
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646140652/
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646154134/
> (x2)
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646156997/
> *
> http://files.apachegeode-ci.info/builds/apache-develop-pr/geode-pr-7407/test-results/repeatTest/1646170346/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10411) XSS vulnerabiltiy in Pulse data browser
[ https://issues.apache.org/jira/browse/GEODE-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615306#comment-17615306 ] ASF subversion and git services commented on GEODE-10411: - Commit 2e689e5c661304c5518d162f3957426014b7ad76 in geode's branch refs/heads/master from Joris Melchior [ https://gitbox.apache.org/repos/asf?p=geode.git;h=2e689e5c66 ] GEODE-10411: fix XSS vulnerability in pulse (#7836) * GEODE-10411: fix XSS vulnerability in pulse - html encode data coming from Geode queries - add cookie parameters to increase browsing security * Fix spotless check errors > XSS vulnerabiltiy in Pulse data browser > --- > > Key: GEODE-10411 > URL: https://issues.apache.org/jira/browse/GEODE-10411 > Project: Geode > Issue Type: Bug > Components: pulse >Affects Versions: 1.12.9, 1.12.10, 1.13.8, 1.13.9, 1.14.4, 1.14.5, 1.15.0, > 1.15.1, 1.16.0 >Reporter: Joris Melchior >Assignee: Joris Melchior >Priority: Major > Labels: pull-request-available > Fix For: 1.12.10, 1.13.9, 1.14.5, 1.15.1, 1.16.0 > > > # Description: > Stored XSS via data injection into Geode database, the injected > payload eventually gets executed on Pulse web application when the > admin querying data from Geode. > # PoC: > Step 1: With Geode up and running, run gfsh command to get into > interactive mode: > shell$ gfsh > Step 2: In gfsh console, execute the following command to insert a > data entry into regionA (assume that regionA is created before). Note > that the value of this data entry contains JavaScript code: > gfsh> put --region=regionA --key="test" --value="alert(1)" > Step 3: Open browser to query editor of Pulse web application at > https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2F192.168.93.153%3A7070%2Fpulse%2FdataBrowser.htmldata=05%7C01%7Cbakera%40vmware.com%7Cc06e6de8d92c4519303708da54fa7d03%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637915732081233095%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=ykaOkxe1hlaE7xl8XQNgBQz2%2Ful1QPxrUChoBkuaeyY%3Dreserved=0 > (assume that already > logged in as admin), execute the following query: > SELECT * FROM /regionA > Step 4: Data from regionA will be retrieved, the XSS payload > eventually get executed > # Why this is an issue? > Developer maybe saves user-controlled data to Geode database, users > maybe submit data via an arbitrary client application (for example, a > web application), the use of gfsh console just simplifies the PoC. > # IMPACT: > Exploiting this XSS vulnerability, an attacker can steal the admin's > session cookie, therefore take over the admin account. > # CVSS: 7.6 HIGH > (https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.first.org%2Fcvss%2Fcalculator%2F3.0%23CVSS%3A3.0%2FAV%3AN%2FAC%3AL%2FPR%3AN%2FUI%3AR%2FS%3AU%2FC%3AH%2FI%3AL%2FA%3ALdata=05%7C01%7Cbakera%40vmware.com%7Cc06e6de8d92c4519303708da54fa7d03%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637915732081233095%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=W5dDA8kMdT1IVeUVX6mhWHhZ2HnAZbXErEB%2F0Tjs5hg%3Dreserved=0 > ) > (re-calculate if not correct) > # Fix: > The Pulse web application must URL encode data retrieved from Geode database. > # Credit: > The issue is found by Nguyen Thai Hung (@nth347), Viettel Cyber Security. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10415) CVEs detected in latest geode
[
https://issues.apache.org/jira/browse/GEODE-10415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615314#comment-17615314
]
ASF subversion and git services commented on GEODE-10415:
-
Commit 5364cce7cdda81d38f8b8275e083d6a870642d51 in geode's branch
refs/heads/master from Mario Kevo
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=5364cce7cd ]
GEODE-10415: bump dependencies due to vulnerability scan (#7855)
> CVEs detected in latest geode
> -
>
> Key: GEODE-10415
> URL: https://issues.apache.org/jira/browse/GEODE-10415
> Project: Geode
> Issue Type: Bug
> Components: build
>Affects Versions: 1.15.0
>Reporter: Shruti
>Assignee: Mario Kevo
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> We are detecting the following CVEs with geode
> High or critical vulnerabilities: 21
> The spring-core is likely Not Affected. But we would like to know about the
> rest of these listed CVEs. Any info is appreciated
> {{ }}
> {{NAME INSTALLED FIXED-IN TYPE
> VULNERABILITY SEVERITY}}
> {{jetty-security 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-server 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-servlet 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-util 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-util-ajax 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-webapp 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-xml 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jgroups 3.6.14.Final 4.0.0
> java-archive GHSA-rc7h-x6cq-988q Critical}}
> {{shiro-cache 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-config-core 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-config-ogdl 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-core 1.9.0 1.9.1
> java-archive GHSA-4cf5-xmhp-3xj7 Critical}}
> {{shiro-core 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-crypto-cipher 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-crypto-core 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-crypto-hash 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-event 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{shiro-lang 1.9.0
> java-archive CVE-2022-32532 Critical}}
> {{spring-core 5.3.20
> java-archive CVE-2016-127 Critical}}
> {{jetty-http 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
> {{jetty-io 9.4.46.v20220331
> java-archive CVE-2022-2048 High}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615332#comment-17615332
]
ASF subversion and git services commented on GEODE-10401:
-
Commit af9bee45f6199743b5dfc0f0195ea91c6ba377e0 in geode-examples's branch
refs/heads/master from Owen Nichols
[ https://gitbox.apache.org/repos/asf?p=geode-examples.git;h=af9bee4 ]
GEODE-10401: Bump version to 1.15.2
As part of the Geode Release Process, the geode-examples build number
must be rolled forward as work begins on the next release
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10412) Destroy region command doesn't clear the region related expired tombstones
[
https://issues.apache.org/jira/browse/GEODE-10412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615324#comment-17615324
]
ASF subversion and git services commented on GEODE-10412:
-
Commit bc47b30c0bb57a9c9437234a8fc8530f1daf14bc in geode's branch
refs/heads/master from Jakov Varenina
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=bc47b30c0b ]
GEODE-10412: Clear expired tombstones during region destroy (#7838)
* GEODE-10412: Clear expired tombstones during region destroy
The issue:
During region destroy operation, the expired tombstones aren't cleared
when non-expired ones are available. Later, these expired
tombstones prevent all other regions' tombstones from being cleared
from memory, causing many issues (memory and disk exhaustion).
The solution:
When a region is destroyed, it must clear all the related expired and
non-expired tombstones from memory.
* Add distributed test that reproduce the issue
* Update after review
> Destroy region command doesn't clear the region related expired tombstones
> --
>
> Key: GEODE-10412
> URL: https://issues.apache.org/jira/browse/GEODE-10412
> Project: Geode
> Issue Type: Bug
> Components: expiration
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> Tombstones in geode are kept on two maps: expiredTombstones and tombstones
> (non-expired ones). When a region is destroyed, it must clear all the related
> expired and non-expired tombstones from memory. Due to the below code bug,
> expired tombstones aren't cleared when non-expired tombstones are available
> during the region destruction:
> {code:java}
> private boolean removeIf(Predicate predicate) {
> return removeUnexpiredIf(predicate) || removeExpiredIf(predicate);
> }
> {code}
> Because of the above, non-expired tombstones are never removed from memory,
> preventing other tombstones from being cleared. Since other tombstones never
> expire, the compaction is not done, and therefore the disk is filled, causing
> the issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10089) release 1.15.0
[ https://issues.apache.org/jira/browse/GEODE-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615310#comment-17615310 ] ASF subversion and git services commented on GEODE-10089: - Commit c5afabc1d98d0c0f1acae65e69bb73523a37 in geode-examples's branch refs/heads/master from Owen Nichols [ https://gitbox.apache.org/repos/asf?p=geode-examples.git;h=c5afab6 ] GEODE-10089: Bump version to 1.15.0 As part of the Geode Release Process, the geode-examples build number must be rolled forward as work begins on the next release > release 1.15.0 > -- > > Key: GEODE-10089 > URL: https://issues.apache.org/jira/browse/GEODE-10089 > Project: Geode > Issue Type: Task > Components: release >Reporter: Owen Nichols >Assignee: Owen Nichols >Priority: Major > Labels: pull-request-available > > As per [Jan 25 dev list > discussion|https://lists.apache.org/thread/s9mpd207h40crcr76fpdfmohdchgdqog], > support/1.15 was cut with the intention of stabilizing and releasing a new > Geode minor. > UPDATE: As per [Mar 16 dev list > discussion|https://lists.apache.org/thread/twlwzcvmx2kqw15whpmxkh2h8bmrok21], > support/1.15 was retracted. Work will continue on develop with an eye toward > re-cutting support/1.15 in ~June > UPDATE: support/1.15 was re-cut on May 9 with an eye toward shipping in mid > to late June (2022). > Release status information is also updated in the Geode [Release > Schedule|https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10089) release 1.15.0
[ https://issues.apache.org/jira/browse/GEODE-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615312#comment-17615312 ] ASF subversion and git services commented on GEODE-10089: - Commit 85c181d2f534286252e10621649d18bd24dfa244 in geode-examples's branch refs/heads/master from Owen Nichols [ https://gitbox.apache.org/repos/asf?p=geode-examples.git;h=85c181d ] Revert "GEODE-10089: Set temporary staging repo" The staging repo no longest exists, so set this back to search the default location (mavencentral) > release 1.15.0 > -- > > Key: GEODE-10089 > URL: https://issues.apache.org/jira/browse/GEODE-10089 > Project: Geode > Issue Type: Task > Components: release >Reporter: Owen Nichols >Assignee: Owen Nichols >Priority: Major > Labels: pull-request-available > > As per [Jan 25 dev list > discussion|https://lists.apache.org/thread/s9mpd207h40crcr76fpdfmohdchgdqog], > support/1.15 was cut with the intention of stabilizing and releasing a new > Geode minor. > UPDATE: As per [Mar 16 dev list > discussion|https://lists.apache.org/thread/twlwzcvmx2kqw15whpmxkh2h8bmrok21], > support/1.15 was retracted. Work will continue on develop with an eye toward > re-cutting support/1.15 in ~June > UPDATE: support/1.15 was re-cut on May 9 with an eye toward shipping in mid > to late June (2022). > Release status information is also updated in the Geode [Release > Schedule|https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615313#comment-17615313
]
ASF subversion and git services commented on GEODE-10401:
-
Commit 4daf50dd17c10302d6da240e5cc93bb362a9d4b5 in geode-examples's branch
refs/heads/master from Mario Kevo
[ https://gitbox.apache.org/repos/asf?p=geode-examples.git;h=4daf50d ]
GEODE-10401: Bump version to 1.15.1
As part of the Geode Release Process, the geode-examples build number
must be rolled forward as work begins on the next release
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10417) Fix NullPointerException when getting events from the gw sender queue to complete transactions
[ https://issues.apache.org/jira/browse/GEODE-10417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615322#comment-17615322 ] ASF subversion and git services commented on GEODE-10417: - Commit 8b751804c5110507b156cc42f4beda713f2fcccd in geode's branch refs/heads/master from Alberto Gomez [ https://gitbox.apache.org/repos/asf?p=geode.git;h=8b751804c5 ] GEODE-10417: Fix NullPointerException in WAN replication (#7845) * GEODE-10417: Fix NullPointerException in WAN replication When the WAN group-transa$ction-events feature is enabled in a parallel gateway sender, it is possible to get a NullPointerException when retrieving events from the queue to complete a transaction if the event in the queue is null. If this situation is reached then the gateway sender dispatcher will not dispatch queue events anymore and therefore the WAN replication will not progress. This happens because the predicates that check if elements in the queue contain a transactionId are not protected against the event being null. A null check has been added before the predicates are invoked so that in case of a null event, the predicate is not invoked and the event is skipped from the checking. * GEODE-10417: Change assertEquals to assertThat > Fix NullPointerException when getting events from the gw sender queue to > complete transactions > -- > > Key: GEODE-10417 > URL: https://issues.apache.org/jira/browse/GEODE-10417 > Project: Geode > Issue Type: Bug > Components: wan >Affects Versions: 1.13.8, 1.14.4, 1.15.0 >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: needsTriage, pull-request-available > Fix For: 1.15.1, 1.16.0 > > > When the WAN group-transaction-events feature is enabled in a parallel > gateway sender, it is possible to get a NullPointerException when retrieving > events from the queue to complete a transaction if the event in the queue is > null. > If this situation is reached then the gateway sender dispatcher will not > dispatch queue events anymore and therefore the WAN replication will not > progress. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10089) release 1.15.0
[ https://issues.apache.org/jira/browse/GEODE-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615309#comment-17615309 ] ASF subversion and git services commented on GEODE-10089: - Commit fef93101383522f1cf69508ec52ac6e8e0d24d80 in geode-examples's branch refs/heads/master from Owen Nichols [ https://gitbox.apache.org/repos/asf?p=geode-examples.git;h=fef9310 ] GEODE-10089: Set temporary staging repo This serves two purposes: it gives the RC pipeline a way to get the nexus staging repo id needed for various tests, and it gives the Jenkins server a valid configuration during the voting period. > release 1.15.0 > -- > > Key: GEODE-10089 > URL: https://issues.apache.org/jira/browse/GEODE-10089 > Project: Geode > Issue Type: Task > Components: release >Reporter: Owen Nichols >Assignee: Owen Nichols >Priority: Major > Labels: pull-request-available > > As per [Jan 25 dev list > discussion|https://lists.apache.org/thread/s9mpd207h40crcr76fpdfmohdchgdqog], > support/1.15 was cut with the intention of stabilizing and releasing a new > Geode minor. > UPDATE: As per [Mar 16 dev list > discussion|https://lists.apache.org/thread/twlwzcvmx2kqw15whpmxkh2h8bmrok21], > support/1.15 was retracted. Work will continue on develop with an eye toward > re-cutting support/1.15 in ~June > UPDATE: support/1.15 was re-cut on May 9 with an eye toward shipping in mid > to late June (2022). > Release status information is also updated in the Geode [Release > Schedule|https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10089) release 1.15.0
[ https://issues.apache.org/jira/browse/GEODE-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615327#comment-17615327 ] ASF subversion and git services commented on GEODE-10089: - Commit e863a993d75b5c392fc2767f2832c2302de385e5 in geode-benchmarks's branch refs/heads/master from Owen Nichols [ https://gitbox.apache.org/repos/asf?p=geode-benchmarks.git;h=e863a993 ] GEODE-10089: Update benchmark baseline Updates the default benchmark baseline on support/1.15 to 1.15.0 > release 1.15.0 > -- > > Key: GEODE-10089 > URL: https://issues.apache.org/jira/browse/GEODE-10089 > Project: Geode > Issue Type: Task > Components: release >Reporter: Owen Nichols >Assignee: Owen Nichols >Priority: Major > Labels: pull-request-available > > As per [Jan 25 dev list > discussion|https://lists.apache.org/thread/s9mpd207h40crcr76fpdfmohdchgdqog], > support/1.15 was cut with the intention of stabilizing and releasing a new > Geode minor. > UPDATE: As per [Mar 16 dev list > discussion|https://lists.apache.org/thread/twlwzcvmx2kqw15whpmxkh2h8bmrok21], > support/1.15 was retracted. Work will continue on develop with an eye toward > re-cutting support/1.15 in ~June > UPDATE: support/1.15 was re-cut on May 9 with an eye toward shipping in mid > to late June (2022). > Release status information is also updated in the Geode [Release > Schedule|https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10420) Handle WAN event when interrupted
[ https://issues.apache.org/jira/browse/GEODE-10420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615323#comment-17615323 ] ASF subversion and git services commented on GEODE-10420: - Commit 62e60b5062f4c164f3a8af0b52fbc1d5e39fb6aa in geode's branch refs/heads/master from Alberto Gomez [ https://gitbox.apache.org/repos/asf?p=geode.git;h=62e60b5062 ] GEODE-10420: Finish distribute() work if interrupted (#7854) It is possible that an event of which a gateway sender is to be notified is lost if during the process the thread is interrupted. The reason is that the distribute() method in the AbstractGatewaySender when it catches the InterruptedException at some point, just returns, but does not put the event in the queue and neither drops it. The fix consists of handling the event correctly (put it in the queue or drop it) if the InterruptedException is caught but when the method returns set again the interrupt flag so that the caller is aware. > Handle WAN event when interrupted > - > > Key: GEODE-10420 > URL: https://issues.apache.org/jira/browse/GEODE-10420 > Project: Geode > Issue Type: Bug > Components: wan >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: needsTriage, pull-request-available > Fix For: 1.15.1, 1.16.0 > > > It is possible that an event of which a gateway sender is to be notified is > lost if during the process the thread is interrupted. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10352) Update Dockerfile to use Ruby >= 2.6 in the tool to preview Geode documentation
[ https://issues.apache.org/jira/browse/GEODE-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615320#comment-17615320 ] ASF subversion and git services commented on GEODE-10352: - Commit 5ee944475a9fc2dd6934276dfcc6b1724f636bcd in geode's branch refs/heads/master from Alberto Gomez [ https://gitbox.apache.org/repos/asf?p=geode.git;h=5ee944475a ] GEODE-10352: Update Ruby version in Geode doc preview tool (#7753) The script to preview the documentation of Geode (./preview-user-guide.sh) is not working anymore. The following error appears while running it (unless you have a previously downloaded geodedocs/temp docker image in your local docker repo): ERROR: Error installing elasticsearch: The last version of faraday (>= 0) to support your Ruby & RubyGems was 1.10.0. Try installing it with `gem install faraday -v 1.10.0` and then running the current command again faraday requires Ruby version >= 2.6. The current ruby version is 2.5.9.229. That error prevents the preview script to work. It is needed to update the docker image used to preview the documentation to use a Ruby version >= 2.6. The versions of other gems also need to be updated after the change of the Ruby version. > Update Dockerfile to use Ruby >= 2.6 in the tool to preview Geode > documentation > --- > > Key: GEODE-10352 > URL: https://issues.apache.org/jira/browse/GEODE-10352 > Project: Geode > Issue Type: Bug > Components: docs >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > The script to preview the documentation of Geode (./preview-user-guide.sh) is > not working anymore. > The following error appears while running it (unless you have a previously > downloaded geodedocs/temp docker image in your local docker repo): > ERROR: Error installing elasticsearch: > The last version of faraday (>= 0) to support your Ruby & RubyGems was > 1.10.0. Try installing it with `gem install faraday -v 1.10.0` and then > running the current command again > faraday requires Ruby version >= 2.6. The current ruby version is > 2.5.9.229. > That error prevents the preview script to work. > It is needed to update the docker image used to preview the documentation to > use a Ruby version >= 2.6 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615333#comment-17615333
]
ASF subversion and git services commented on GEODE-10401:
-
Commit 6f1009f573d7e0058a79253890300b973e14eafa in geode-benchmarks's branch
refs/heads/support/1.15 from Owen Nichols
[ https://gitbox.apache.org/repos/asf?p=geode-benchmarks.git;h=6f1009f5 ]
GEODE-10401: Bump version to 1.15.2
As part of the Geode Release Process, the geode-benchmarks build number
must be rolled forward as work begins on the next release
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10281) Internal conflict replicated region resolution causes data inconsistency between two sites
[
https://issues.apache.org/jira/browse/GEODE-10281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615308#comment-17615308
]
ASF subversion and git services commented on GEODE-10281:
-
Commit fdb5e92d93095dc26f82e2935cce18e121f70b36 in geode's branch
refs/heads/master from Jakov Varenina
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=fdb5e92d93 ]
GEODE-10281: Fix WAN data inconsistency (#7665)
> Internal conflict replicated region resolution causes data inconsistency
> between two sites
> --
>
> Key: GEODE-10281
> URL: https://issues.apache.org/jira/browse/GEODE-10281
> Project: Geode
> Issue Type: Bug
> Components: wan
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}It seems that the following PR
> {color}[{color:#4a6ee0}[https://github.com/apache/geode/pull/3044]{color}]{color:#0e101a}
> has not taken into consideration that event with a flag
> isConcurrencyConflict=true could delete the valid event from the queue due to
> conflation.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10403) Distributed deadlock when stopping gateway sender
[
https://issues.apache.org/jira/browse/GEODE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615321#comment-17615321
]
ASF subversion and git services commented on GEODE-10403:
-
Commit 3ada8fe80f7569874f643c32d585658e98f4615a in geode's branch
refs/heads/master from Alberto Gomez
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=3ada8fe80f ]
GEODE-10403: Fix distributed deadlock with stop gw sender (#7830)
There is a distributed deadlock that can appear
when stopping the gateway sender if a race condition
happens in which the stop gateway sender command gets blocked
indefinitely trying to get the size of the queue from remote peers
(ParallelGatewaySenderQueue.size() call) and
also one call to store one event in the queue tries to get
the lifecycle lock (acquired by the gateway sender command).
These two calls could get into a deadlock under heavy load and
make the system unresponsive for any traffic request (get, put, ...).
In order to avoid it, in the storage of the event in the gateway
sender queue (AbstractGatewaySender.distribute() call),
instead to trying to get the lifecycle lock without
any timeout, a try with a timeout is added. If the
try returns false it is checked if the gateway sender is running. If
it is not running, the event is dropped and there is no need to get the lock.
Otherwise, the lifecycle lock acquire is retried until it succeeds or
the gateway sender is stopped.
> Distributed deadlock when stopping gateway sender
> -
>
> Key: GEODE-10403
> URL: https://issues.apache.org/jira/browse/GEODE-10403
> Project: Geode
> Issue Type: Bug
> Components: wan
>Affects Versions: 1.12.9, 1.13.8, 1.14.4, 1.15.0
>Reporter: Alberto Gomez
>Assignee: Alberto Gomez
>Priority: Major
> Labels: needsTriage, pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> A distributed deadlock has been found during some tests of a Geode system
> with WAN replication when stopping the gateway sender while sending a fair
> amount of operations to the servers.
> The distributed deadlock manifests in the gateway sender stop command hanging
> forever and by all normal Geode operations from clients (gets, puts,...) not
> being responded.
> The situation is provoked by the Gateway sender stop command that first takes
> the lifecycle lock and then, at a given point, tries to retrieve the size of
> the gateway sender. This operation, that requires communication with the
> other peers never finishes, probably because the response from one of the
> peers is never received.
> Another thread is blocked when trying to acquire the lifecycle lock in
> AbstractGatewaySender.distribute().
> Finally many threads handling Geode operations (get, put...) get blocked in
> the DistributedCacheOperation._distribute() call waiting for a response from
> another peer.
> Thread dump section from blocked gateway sender stop command in call to get
> size of queue:
> {{"ConcurrentParallelGatewaySenderEventProcessor Stopper Thread1" #1316
> daemon prio=10 os_prio=0 cpu=45.55ms elapsed=4152.76s tid=0x7f92bc1c2000
> nid=0x2154 waiting on condition [0x7f9179cd2000]}}
> {{ java.lang.Thread.State: TIMED_WAITING (parking)}}
> {{ at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)}}
> {{ - parking to wait for <0x00031ca2be50> (a
> java.util.concurrent.CountDownLatch$Sync)}}
> {{ at
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.11/LockSupport.java:234)}}
> {{ at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(java.base@11.0.11/AbstractQueuedSynchronizer.java:1079)}}
> {{ at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(java.base@11.0.11/AbstractQueuedSynchronizer.java:1369)}}
> {{ at
> java.util.concurrent.CountDownLatch.await(java.base@11.0.11/CountDownLatch.java:278)}}
> {{ at
> org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)}}
> {{ at
> org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:731)}}
> {{ at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:802)}}
> {{ at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:779)}}
> {{ at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:865)}}
> {{ at
> org.apache.geode.internal.cache.partitioned.SizeMessage$SizeResponse.waitBucketSizes(SizeMessage.java:344)}}
> {{ at
>
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615329#comment-17615329
]
ASF subversion and git services commented on GEODE-10401:
-
Commit 006c58761691409ac3e7ba57be74af3134fd2fde in geode's branch
refs/heads/support/1.15 from Owen Nichols
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=006c587616 ]
GEODE-10401: Replace 1.15.0 with 1.15.1 as old version
Replace 1.15.0 with 1.15.1 in old versions on support/1.15
to enable rolling upgrade tests from 1.15.1
The serialization version has not changed between 1.15.0 and 1.15.1,
so there should be no need to keep both
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10348) Correct documentation about conflation
[ https://issues.apache.org/jira/browse/GEODE-10348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615319#comment-17615319 ] ASF subversion and git services commented on GEODE-10348: - Commit b64cdd1473de46ec32064e06fb7349f499347676 in geode's branch refs/heads/master from Alberto Gomez [ https://gitbox.apache.org/repos/asf?p=geode.git;h=b64cdd1473 ] GEODE-10348: Fix documentation on conflation (#7746) The Geode documentation states on conflation: "When an update is added to a queue that has conflation enabled, if there is already an update message in the queue for the entry key, then the existing message assumes the value of the new update and the new update is dropped, as shown here for key A." Nevertheless, that is not correct. The actual behavior is the following: "When an update is added to a queue that has conflation enabled, if there is already an update message in the queue for the entry key, then the existing message is removed and the new update is added to the end of the queue, as shown here for key A." The text has been updated as well as the the figure with the example. > Correct documentation about conflation > -- > > Key: GEODE-10348 > URL: https://issues.apache.org/jira/browse/GEODE-10348 > Project: Geode > Issue Type: Bug > Components: docs, wan >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > The Geode documentation states on conflation: > "When an update is added to a queue that has conflation enabled, if there is > already an update message in the queue for the entry key, then the existing > message assumes the value of the new update and the new update is dropped, as > shown here for key A." > Nevertheless, that is not correct. The actual behavior is the following: > "When an update is added to a queue that has conflation enabled, if there is > already an update message in the queue for the entry key, then the existing > message is removed and the new update is added to the end of the queue, as > shown here for key A." -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10346) Correct batch-time-interval description in documentation
[ https://issues.apache.org/jira/browse/GEODE-10346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615318#comment-17615318 ] ASF subversion and git services commented on GEODE-10346: - Commit 982333d4ba532c922ac9f176e72fb6816f68ddb8 in geode's branch refs/heads/master from Alberto Gomez [ https://gitbox.apache.org/repos/asf?p=geode.git;h=982333d4ba ] GEODE-10346: Correct description of batch-time-interval in doc. (#7742) > Correct batch-time-interval description in documentation > > > Key: GEODE-10346 > URL: https://issues.apache.org/jira/browse/GEODE-10346 > Project: Geode > Issue Type: Bug > Components: docs, wan >Reporter: Alberto Gomez >Assignee: Alberto Gomez >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > The description of the batch-time-interval parameter for the Gateway Sender > in the Geode documentation states the following: > "Maximum amount of time, in ms, that can elapse before a batch is delivered." > Nevertheless, that is not completely true. > The number of ms that can elapse before a batch is delivered could be longer > than what is configured in batch-time-interval in the case that the batch > being created has not yet reached the value of max-batch-size and there are > still events in the gateway sender's queue to be added to the batch. If that > is the case, new events will keep being added to the batch without taking > into account the value for batch-time-interval. > The batch-time-interval parameter is only used when the batch being filled > has not reached the max-batch-size but there are no events in the queue. In > that case, in order not to delay the delivery of the batch until there are > events in the queue, this value is used to determine if the batch must be > sent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10089) release 1.15.0
[ https://issues.apache.org/jira/browse/GEODE-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615304#comment-17615304 ] ASF subversion and git services commented on GEODE-10089: - Commit 2fe3ea0e077717a36aaee94666c97a3b3d6b349d in geode's branch refs/heads/master from Owen Nichols [ https://gitbox.apache.org/repos/asf?p=geode.git;h=2fe3ea0e07 ] GEODE-10089: Bump version to 1.15.1 As part of the Geode Release Process, the build number must be rolled forward so work can begin on the next release > release 1.15.0 > -- > > Key: GEODE-10089 > URL: https://issues.apache.org/jira/browse/GEODE-10089 > Project: Geode > Issue Type: Task > Components: release >Reporter: Owen Nichols >Assignee: Owen Nichols >Priority: Major > Labels: pull-request-available > > As per [Jan 25 dev list > discussion|https://lists.apache.org/thread/s9mpd207h40crcr76fpdfmohdchgdqog], > support/1.15 was cut with the intention of stabilizing and releasing a new > Geode minor. > UPDATE: As per [Mar 16 dev list > discussion|https://lists.apache.org/thread/twlwzcvmx2kqw15whpmxkh2h8bmrok21], > support/1.15 was retracted. Work will continue on develop with an eye toward > re-cutting support/1.15 in ~June > UPDATE: support/1.15 was re-cut on May 9 with an eye toward shipping in mid > to late June (2022). > Release status information is also updated in the Geode [Release > Schedule|https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10385) User Guide: Remove bad G1 tuning advice
[ https://issues.apache.org/jira/browse/GEODE-10385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615305#comment-17615305 ] ASF subversion and git services commented on GEODE-10385: - Commit 8cc780a557ad9052a27c12de00e18ac0bdf1016c in geode's branch refs/heads/master from Dave Barnes [ https://gitbox.apache.org/repos/asf?p=geode.git;h=8cc780a557 ] GEODE-10385: User Guide - Remove bad G1GC tuning advice (#7815) > User Guide: Remove bad G1 tuning advice > --- > > Key: GEODE-10385 > URL: https://issues.apache.org/jira/browse/GEODE-10385 > Project: Geode > Issue Type: Improvement > Components: docs >Affects Versions: 1.14.4 >Reporter: Dave Barnes >Assignee: Dave Barnes >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.15.1 > > > The "Managing Heap Memory" page provides some G1 tuning advice that is > actually detrimental to performance. The whole section needs some work, but > this passage, in particular, is wrong: > If you find the Apache Geode Resource Manager does not detect crossing the > eviction or critical threshold quickly enough, it has been observed that its > responsiveness is increased by reducing the default value of > --J-XX:MaxGCPauseMillis=VALUE JVM parameter (which is 200). Be sure to take > into account that this change also increases the amount of time spent in > garbage collection. > Cause: This advice was originally intended only to maximize performance of a > specific test case. It is NOT good advice for production systems. > https://geode.apache.org/docs/guide/114/managing/heap_use/heap_management.html > Short-term solution: delete the passage. > Long-term solution: Come up with some better ideas. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10422) Add doc for using parallel recovery disk store
[ https://issues.apache.org/jira/browse/GEODE-10422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615311#comment-17615311 ] ASF subversion and git services commented on GEODE-10422: - Commit 3d62f0fff21329a319661780f143bb7683135fe4 in geode's branch refs/heads/master from Mario Kevo [ https://gitbox.apache.org/repos/asf?p=geode.git;h=3d62f0fff2 ] GEODE-10422: add Note for parallel recovery disk store (#7858) * GEODE-10422: add Note for parallel recovery disk store * empty commit to re-launch CI > Add doc for using parallel recovery disk store > -- > > Key: GEODE-10422 > URL: https://issues.apache.org/jira/browse/GEODE-10422 > Project: Geode > Issue Type: Improvement > Components: docs >Affects Versions: 1.14.0, 1.15.0 >Reporter: Mario Kevo >Assignee: Mario Kevo >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > There are missing part for parallel recovery of disk stores. > It isn't specified that in case the user use the same disk store for the pdx > and the region, disk store recovery will go in the sequential mode. So in > case you want to use parallel disk store recovery it should be different disk > stores for the pdx and the disk store. Otherwise, it will go sequentially, > despite of default behavior to run in the parallel mode. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-9484) Data inconsistency in replicated region with 3 or more servers, and one server is down
[ https://issues.apache.org/jira/browse/GEODE-9484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615307#comment-17615307 ] ASF subversion and git services commented on GEODE-9484: Commit 7838d5d5210ab3afe243b2045242d7bfd6acc6ed in geode's branch refs/heads/master from Mario Kevo [ https://gitbox.apache.org/repos/asf?p=geode.git;h=7838d5d521 ] GEODE-9484: Improve sending message to multy destinations (#7664) (#7853) Co-authored-by: Mario Ivanac <48509724+miva...@users.noreply.github.com> > Data inconsistency in replicated region with 3 or more servers, and one > server is down > --- > > Key: GEODE-9484 > URL: https://issues.apache.org/jira/browse/GEODE-9484 > Project: Geode > Issue Type: Improvement > Components: client/server, regions >Affects Versions: 1.13.0 >Reporter: Mario Ivanac >Assignee: Mario Ivanac >Priority: Major > Labels: pull-request-available > Fix For: 1.15.1, 1.16.0 > > > We have configured replicated region with 3 or more servers, and client is > configured with read timeout set to value same or smaller than member timeout. > In case while client is putting data in region, one of replicated servers is > shutdown, it is observed that we have data inconsistency. > > We see that data part of data is written in server connected with client, but > in remaining replicated servers it is missing. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10089) release 1.15.0
[ https://issues.apache.org/jira/browse/GEODE-10089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615302#comment-17615302 ] ASF subversion and git services commented on GEODE-10089: - Commit dc21e1c0dd44e252d77b08dc035fc1cdf9ece78a in geode's branch refs/heads/master from Owen Nichols [ https://gitbox.apache.org/repos/asf?p=geode.git;h=dc21e1c0dd ] GEODE-10089: update Dockerfile to apache-geode 1.15.0 The Dockerfile is updated _after_ the release is already tagged, because it needs to embed the sha256 of the release > release 1.15.0 > -- > > Key: GEODE-10089 > URL: https://issues.apache.org/jira/browse/GEODE-10089 > Project: Geode > Issue Type: Task > Components: release >Reporter: Owen Nichols >Assignee: Owen Nichols >Priority: Major > Labels: pull-request-available > > As per [Jan 25 dev list > discussion|https://lists.apache.org/thread/s9mpd207h40crcr76fpdfmohdchgdqog], > support/1.15 was cut with the intention of stabilizing and releasing a new > Geode minor. > UPDATE: As per [Mar 16 dev list > discussion|https://lists.apache.org/thread/twlwzcvmx2kqw15whpmxkh2h8bmrok21], > support/1.15 was retracted. Work will continue on develop with an eye toward > re-cutting support/1.15 in ~June > UPDATE: support/1.15 was re-cut on May 9 with an eye toward shipping in mid > to late June (2022). > Release status information is also updated in the Geode [Release > Schedule|https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10386) Document JDK 17 for 1.15 release
[ https://issues.apache.org/jira/browse/GEODE-10386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615301#comment-17615301 ] ASF subversion and git services commented on GEODE-10386: - Commit ab5e533494787922a22d278ae01d87c1d031e212 in geode's branch refs/heads/master from Dave Barnes [ https://gitbox.apache.org/repos/asf?p=geode.git;h=ab5e533494 ] GEODE-10386:Document JDK 17 for 1.15 release (#7813) Co-authored by: @animatedmax > Document JDK 17 for 1.15 release > > > Key: GEODE-10386 > URL: https://issues.apache.org/jira/browse/GEODE-10386 > Project: Geode > Issue Type: Improvement > Components: docs >Affects Versions: 1.15.0 >Reporter: Dave Barnes >Assignee: Max Hufnagel >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.15.1 > > > Document JDK 17 for 1.5 release -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10389) Document upgrade to Geode 1.15 with SSL protocol workaround
[ https://issues.apache.org/jira/browse/GEODE-10389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615300#comment-17615300 ] ASF subversion and git services commented on GEODE-10389: - Commit ececd27a039dc7dd1eb836d09532995782486869 in geode's branch refs/heads/master from Dave Barnes [ https://gitbox.apache.org/repos/asf?p=geode.git;h=ececd27a03 ] GEODE-10389: Document upgrade to Geode 1.15 with SSL protocol workaround (#7812) > Document upgrade to Geode 1.15 with SSL protocol workaround > --- > > Key: GEODE-10389 > URL: https://issues.apache.org/jira/browse/GEODE-10389 > Project: Geode > Issue Type: Sub-task > Components: docs >Affects Versions: 1.15.0 >Reporter: Dave Barnes >Assignee: Dave Barnes >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.15.1 > > > Document migration steps to mitigate protocol setting mismatch (GEODE-9991) > when upgrading to v1.5 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615297#comment-17615297
]
ASF subversion and git services commented on GEODE-10401:
-
Commit 40c7c10b035ac6069c398ec102a4daa3d759a24f in geode-examples's branch
refs/heads/support/1.15 from Owen Nichols
[ https://gitbox.apache.org/repos/asf?p=geode-examples.git;h=40c7c10 ]
Revert "GEODE-10401: Set temporary staging repo"
The staging repo no longest exists, so set this back to search the
default location (mavencentral)
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10390) User guide: update authentication expiry instructions
[ https://issues.apache.org/jira/browse/GEODE-10390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615299#comment-17615299 ] ASF subversion and git services commented on GEODE-10390: - Commit 81ae2db3e54f093cb7d0bbcc6d462d4cf6141873 in geode's branch refs/heads/master from Dave Barnes [ https://gitbox.apache.org/repos/asf?p=geode.git;h=81ae2db3e5 ] GEODE-10390: User guide: update authentication expiry instructions (#7809) > User guide: update authentication expiry instructions > - > > Key: GEODE-10390 > URL: https://issues.apache.org/jira/browse/GEODE-10390 > Project: Geode > Issue Type: Improvement > Components: docs >Affects Versions: 1.15.0 >Reporter: Dave Barnes >Assignee: Dave Barnes >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.15.1 > > > In "Implementing Authentication" indicate that "A SecurityManager > implementation that supports reauthentication using expiring credentials must > also support non-expiring credentials for cluster members” > In "Implementing Authentication Expiry", indicate that "Authentication expiry > is supported only with client connections." -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10388) srcDist includes local plugin binaries
[ https://issues.apache.org/jira/browse/GEODE-10388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615298#comment-17615298 ] ASF subversion and git services commented on GEODE-10388: - Commit f4c05ad2c15534bede369699f778d92c1a5a833d in geode's branch refs/heads/master from Robert Houghton [ https://gitbox.apache.org/repos/asf?p=geode.git;h=f4c05ad2c1 ] GEODE-10388: create better output filter for srcDist task (#7806) Replace the brittle exclude list for `build` and `out` directories with a closure containing comparison logic for file-type and name. Makes sure that regular files named `out` are still archived. (cherry picked from commit bbf51e74fccf59cbdf6a7ca283342a09ec9bffb3) > srcDist includes local plugin binaries > -- > > Key: GEODE-10388 > URL: https://issues.apache.org/jira/browse/GEODE-10388 > Project: Geode > Issue Type: Bug > Components: build >Affects Versions: 1.15.0, 1.16.0 >Reporter: Robert Houghton >Assignee: Robert Houghton >Priority: Major > Labels: needsTriage, pull-request-available > Fix For: 1.15.0, 1.16.0 > > > After refactoring buildSrc into buildTools, the exclusion rule for the > `srcDist` task fails to filter out the `build` output directory of the plugin. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615290#comment-17615290
]
ASF subversion and git services commented on GEODE-10401:
-
Commit 10582d0fc7741eb50169fa78f59fc686f4394d66 in geode's branch
refs/heads/support/1.15 from Owen Nichols
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=10582d0fc7 ]
GEODE-10401: update Dockerfile to apache-geode 1.15.1
The Dockerfile is updated _after_ the release is already tagged,
because it needs to embed the sha256 of the release
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10401) Oplog recovery takes too long due to fault in fastutil library
[
https://issues.apache.org/jira/browse/GEODE-10401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615291#comment-17615291
]
ASF subversion and git services commented on GEODE-10401:
-
Commit 7cae66172f950c779625c241553edcaecd424d62 in geode-native's branch
refs/heads/support/1.15 from Owen Nichols
[ https://gitbox.apache.org/repos/asf?p=geode-native.git;h=7cae66172 ]
GEODE-10401: Update Dockerfile and vars
Native client hardcodes Geode version to test with in several places.
Update native Dockerfile and other variables to apache-geode 1.15.1
> Oplog recovery takes too long due to fault in fastutil library
> --
>
> Key: GEODE-10401
> URL: https://issues.apache.org/jira/browse/GEODE-10401
> Project: Geode
> Issue Type: Bug
>Reporter: Jakov Varenina
>Assignee: Jakov Varenina
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.1, 1.16.0
>
>
> {color:#0e101a}As we already know, the .drf file delete operations only
> contain OplogEntryID. During recovery, the server reads (byte by byte) each
> OplogEntryID and stores it in a HashSet to use later when recovering .crf
> files. There are two types of HashSets: IntOpenHashSet and LongOpenHashSet.
> The OplogEntryID of type
> {color}_{color:#0e101a}integer{color}_{color:#0e101a} will be stored in
> IntOpenHashSet, and {color}_{color:#0e101a}long
> integer{color}_{color:#0e101a} in LongOpenHashSet, probably due to memory
> optimization and performance factors. OplogEntryID starts with a zero and
> increments throughout time.
> {color}
> {color:#0e101a}We have observed in logs that between exception (There is a
> large number of deleted entries) and the previous log have passed more than 4
> minutes (sometimes even more).{color}
> {code:java}
> {"timestamp":"2022-06-14T21:41:43.772+08:00","severity":"info","message":"Recovering
> oplog#271 /opt/dbservice/data/datastore/BACKUPdataDiskStore_271.drf for disk
> store dataDiskStore.","metadata":
> {"timestamp":"2022-06-14T21:46:02.152+08:00","severity":"warning","message":"There
> is a large number of deleted entries within the disk-store, please execute
> an offline
> compaction.","metadata":
> {code}
> {color:#0e101a}When the above exception occurs, that means that the limit of
> {color}_{color:#0e101a}805306401{color}_{color:#0e101a} entries in
> IntOpenHashSet has been reached. In that case, the server rolls to the new
> IntOpenHashSet, where an exception and the delay could happen again.{color}
> {color:#0e101a}The problem is that due to the fault in FastUtil dependency
> (IntOpenHashSet and LongOpenHashSet), the unnecessary rehashing happens
> multiple times before the max size is reached. The{color}
> _{color:#0e101a}rehashing starts from{color}_ {color:#0e101a}805306368
> onwards for each new entry until the max size. This rehashing adds several
> minutes to .drf Oplog recovery, but does nothing as max is already
> reached.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (GEODE-10423) Document the system property “ON_DISCONNECT_CLEAR_PDXTYPEIDS“
[ https://issues.apache.org/jira/browse/GEODE-10423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614928#comment-17614928 ] ASF subversion and git services commented on GEODE-10423: - Commit 018f2b4a8531939ad7500130bb37d6bae73455cf in geode's branch refs/heads/develop from Tim Zhang [ https://gitbox.apache.org/repos/asf?p=geode.git;h=018f2b4a85 ] GEODE-10423: Document the system property “ON_DISCONNECT_CLEAR_PDXTYP… (#7861) * GEODE-10423: Document the system property “ON_DISCONNECT_CLEAR_PDXTYPEIDS“ Document the java system property “ON_DISCONNECT_CLEAR_PDXTYPEIDS“. This property is used by Java client, add instructions for using this property. * GEODE-10423: Changes after review * GEODE-10423: Changes after review * GEODE-10423: Changes after review * GEODE-10423: change "Applies to (Java Client, Native Client)" to "Client type" > Document the system property “ON_DISCONNECT_CLEAR_PDXTYPEIDS“ > - > > Key: GEODE-10423 > URL: https://issues.apache.org/jira/browse/GEODE-10423 > Project: Geode > Issue Type: Improvement > Components: docs >Reporter: Tim Zhang >Assignee: Tim Zhang >Priority: Minor > Labels: pull-request-available > > Document the java system property “ON_DISCONNECT_CLEAR_PDXTYPEIDS“, this > property is used by Java client. add instructions for using this property. -- This message was sent by Atlassian Jira (v8.20.10#820010)
