Re: Cache expiry policy not deleting records from disk(native persistence)
Shiva, Hopefully, someone from the dev community will pick this ticket up soon and solve the task. In the meantime, Artem, would you mind documenting this limitation referring to ticket 10862? - Denis On Tue, Sep 10, 2019 at 12:50 AM Shiva Kumar wrote: > I have filed a bug https://issues.apache.org/jira/browse/IGNITE-12152 but > this is same as https://issues.apache.org/jira/browse/IGNITE-10862 > Any idea on the timeline of these tickets? > In the documentation > https://apacheignite.readme.io/v2.7/docs/expiry-policies > it says when native persistence is enabled "*expired entries are removed > from both memory and disk tiers*" but in the disk it just mark the pages > as unwanted pages and same disk space used by these unwanted pages will be > used to store new pages but it will not remove unwanted pages from disk and > so it will not release disk space used by these unwanted pages. > > here is the developer's discussion link > > http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html > > > On Mon, Sep 9, 2019 at 11:53 PM Shiva Kumar > wrote: > >> Hi >> I have deployed ignite on kubernetes and configured two seperate >> persistent volume for WAL and persistence. >> The issue Iam facing is same as >> https://issues.apache.org/jira/browse/IGNITE-10862 >> >> Thanks >> Shiva >> >> On Mon, 9 Sep, 2019, 10:47 PM Andrei Aleksandrov, < >> aealexsand...@gmail.com> wrote: >> >>> Hello, >>> >>> I guess that generated WAL will take this disk space. Please read about >>> WAL here: >>> >>> https://apacheignite.readme.io/docs/write-ahead-log >>> >>> Please provide the size of every folder under /opt/ignite/persistence. >>> >>> BR, >>> Andrei >>> 9/6/2019 9:45 PM, Shiva Kumar пишет: >>> >>> Hi all, >>> I have set cache expiry policy like this >>> >>> >>> >>> >>> >>> >> class="org.apache.ignite.configuration.CacheConfiguration"> >>> >>> >>> >>> >>> >>> >> factory-method="factoryOf"> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> And batch inserting records to one of the table which is created with >>> above cache template. >>> Around 10 minutes, I ingested ~1.5GB of data and after 10 minutes >>> records started reducing(expiring) when I monitored from sqlline. >>> >>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from >>> DIMENSIONS; >>> >>> >>> COUNT(ID) >>> >>> >>> 248896 >>> >>> 1 row selected (0.86 seconds) >>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from >>> DIMENSIONS; >>> >>> >>> COUNT(ID) >>> >>> >>> 222174 >>> >>> 1 row selected (0.313 seconds) >>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from >>> DIMENSIONS; >>> >>> >>> COUNT(ID) >>> >>> >>> 118154 >>> >>> 1 row selected (0.15 seconds) >>> 0: jdbc:ignite:thin://192.168.*.*:10800> >>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from >>> DIMENSIONS; >>> >>> >>> COUNT(ID) >>> >>> >>> 76061 >>> >>> 1 row selected (0.106 seconds) >>> 0: jdbc:ignite:thin://192.168.*.*:10800> >>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from >>> DIMENSIONS; >>> >>> >>> COUNT(ID) >>> >>> >>> 41671 >>> >>> 1 row selected (0.063 seconds) >>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from >>> DIMENSIONS; >>> >>> >>> COUNT(ID) >>> >>> >>> 18455 >>> >>> 1 row selected (0.037 seconds) >>> 0: jdbc:ignite:thin://192.168.*.*:10800> select count(ID) from >>> DIMENSIONS; >>> >>> >>> COUNT(ID) >>> >>> >>> 0 >>> >>> 1 row selected (0.014 seconds) >>> >>> >>> But in the meantime, the disk space used by the persistence store was in >>> the same usage level instead of decreasing. >>> >>> >>> [ignite@ignite-cluster-ign-shiv-0 ignite]$ while true ; do df -h >>> /opt/ignite/persistence/; sleep 1s; done >>> Filesystem Size Used Avail Use% Mounted on >>> /dev/vdj 15G 1.6G 14G 11% /opt/ignite/persistence >>> Filesystem Size Used Avail Use% Mounted on
[jira] [Created] (IGNITE-12167) I think we should add logic for clearing custom pageHndWrapper in afterTestsStopped() in GridCommonAbstractTest or GridAbstractTest https://ggtc.gridgain.com/viewLog.ht
Denis Chudov created IGNITE-12167:
-
Summary: I think we should add logic for clearing custom
pageHndWrapper in afterTestsStopped() in GridCommonAbstractTest or
GridAbstractTest https://ggtc.gridgain.com/viewLog.html?buildId=2364333
Key: IGNITE-12167
URL: https://issues.apache.org/jira/browse/IGNITE-12167
Project: Ignite
Issue Type: Improvement
Reporter: Denis Chudov
Assignee: Denis Chudov
We should move logic for clearing custom pageHndWrapperin
{{afterTestsStopped()}} in {{GridCommonAbstractTest}} or {{GridAbstractTest to
ensure that we will not have problems in any tests which set custom wrapper}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
Re: [DISCUSSION] REST requests explicit authorization.
Hello, Mikhail! Why do we need to avoid tough mapping GridRestCommand -> SecurityPermission? Maybe it would be more transparent if we add to the GridRestCommand a field that will contain SecurityPermission for this command? ср, 11 сент. 2019 г. в 17:34, Mikhail Petrov : > Igniters, > > I would like to suggest expanding the IgniteSecurity interface with a > method for REST requests explicit authorization (e.g. public void > authorize(GridRestRequest req) throws SecurityException;). > > Currently, REST request authorization starts in > GridRestProcessor#authorize(GridRestRequest) where GridRestCommand is > converted to SecurityPermission and then passed to > IgniteSecurity#authorize(String, SecurityPermission) for final > authorization. > > I propose to allow GridSecurityProcessor to make an authorization > decision on its own by giving it GridRestRequest. > > This approach can help to avoid tough mapping GridRestCommand -> > SecurityPermission and achieve much more flexibility in tweaking REST > request authorization. > > I will appreciate your feedback on this proposal. > >
[jira] [Created] (IGNITE-12166) .NET: ExecutableTest fails with PathTooLong error
Pavel Tupitsyn created IGNITE-12166: --- Summary: .NET: ExecutableTest fails with PathTooLong error Key: IGNITE-12166 URL: https://issues.apache.org/jira/browse/IGNITE-12166 Project: Ignite Issue Type: Bug Components: platforms Reporter: Pavel Tupitsyn Assignee: Pavel Tupitsyn When repo root path is long enough, it causes ExecutableTests to fail - classpath exceeds Win32 limit of 32767 characters. If fact, we don't need to pass classpath explicitly to IgniteProcess. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IGNITE-12165) Negative time in Transaction time dump
Denis Chudov created IGNITE-12165:
-
Summary: Negative time in Transaction time dump
Key: IGNITE-12165
URL: https://issues.apache.org/jira/browse/IGNITE-12165
Project: Ignite
Issue Type: Improvement
Reporter: Denis Chudov
Assignee: Denis Chudov
After implementing ticket https://ggsystems.atlassian.net/browse/GG-21272 we
have transaction dumps in the logs.
there are some issues with information in these dumps:
{code:java}
[11:53:36,154][INFO][snapshot-scheduler-restats-#69][GridNearTxLocal]
Transaction time dump [startTime=11:53:36.081, totalTime=65, systemTime=3,
userTime=62, cacheOperationsTime=-41943785508, rollbackTime=41943785512,
tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [], nearLocallyMapped=false,
colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=true,
trackTimeout=false, systemTime=3953576, systemStartTime=0, prepareStartTime=0,
prepareTime=0, commitOrRollbackStartTime=0,
commitOrRollbackTime=41943785512318555,
txDumpsThrottling=org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager$TxDumpsThrottling@760d8025,
lb=null, thread=snapshot-scheduler-restats-#69, mappings=IgniteTxMappingsImpl
[], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false,
nearNodes=KeySetView [], dhtNodes=KeySetView [], explicitLock=false,
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
depEnabled=false, txState=IgniteTxStateImpl [activeCacheIds=[-2100569601],
recovery=false, txMap=HashSet [IgniteTxEntry [key=KeyCacheObjectImpl [part=2,
val=SnapshotScheduleKey [id=_SCHEDULES_], hasValBytes=true],
cacheId=-2100569601, txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=2,
val=SnapshotScheduleKey [id=_SCHEDULES_], hasValBytes=true],
cacheId=-2100569601], val=[op=READ, val=null], prevVal=[op=NOOP, val=null],
oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1,
conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null,
filters=null, filtersPassed=false, filtersSet=true, entry=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=2, val=SnapshotScheduleKey [id=_SCHEDULES_],
hasValBytes=true], val=null, ver=GridCacheVersion [topVer=0, order=0,
nodeOrder=0], hash=886348002, extras=null, flags=0]GridDistributedCacheEntry
[super=]GridDhtDetachedCacheEntry [super=], prepared=0, locked=true,
nodeId=1adbae78-40fe-480d-803d-4d498919ae63, locMapped=false, expiryPlc=null,
transferExpiryPlc=false, flags=0, partUpdateCntr=0, serReadVer=null,
xidVer=GridCacheVersion [topVer=179672005, order=1568192005565,
nodeOrder=1, super=IgniteTxAdapter [xidVer=GridCacheVersion
[topVer=179672005, order=1568192005565, nodeOrder=1], writeVer=null,
implicit=false, loc=true, threadId=125, startTime=1568192016081,
nodeId=9e00ae45-4084-4f29-949f-c19c933f4299, startVer=GridCacheVersion
[topVer=179672005, order=1568192005565, nodeOrder=1], endVer=null,
isolation=REPEATABLE_READ, concurrency=PESSIMISTIC, timeout=0,
sysInvalidate=false, sys=true, plc=5, commitVer=null, finalizing=NONE,
invalidParts=null, state=ROLLED_BACK, timedOut=false,
topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], duration=60ms,
onePhaseCommit=false], size=1
{code}
For example:
# Negative time: cacheOperationsTime=-41943785508,
# Huge times: rollbackTime=41943785512
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
