[
https://issues.apache.org/jira/browse/APEXMALHAR-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bright chen updated APEXMALHAR-2332:
------------------------------------
Remaining Estimate: 120h (was: 48h)
Original Estimate: 120h (was: 48h)
Description:
Current StateTracker free memory was triggered by a timer. The default the
timer value was DAGContext.STREAMING_WINDOW_SIZE_MILLIS.defaultValue *
OperatorContext.APPLICATION_WINDOW_COUNT.defaultValue. It would have memory
leak if the process with operator thread and memory release thread as following:
bucket1: put(), put() ... put()
bucket2: put(), put() ... put()
freeMemory(): {bucket removed from bucketHeap and bucketAccessTimes}
commit: bucket1, bucket2
in this case, nothing was freed and the bucket can't be freed any more
And the default value of free memory could large and memory used up even before
get chance of free memory.
was:
StateTracker#bucketAccessTimes keep the bucket access time order by access
time. It was used by free memory thread to decide which bucket can be freed.
So, each access to bucket include put and get should update the access time.
As bucketAccessTimes and bucketHeap are shared by two thread. update them for
each operation could impact the performance. It better to update period. As
Bucket don't support window operation. I am going to keep the update time and
update when time out.
Summary: StateTracker should free memory after committed (was:
StateTracker#bucketAccessed should be called each time access to the bucket)
> StateTracker should free memory after committed
> -----------------------------------------------
>
> Key: APEXMALHAR-2332
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2332
> Project: Apache Apex Malhar
> Issue Type: Bug
> Reporter: bright chen
> Assignee: bright chen
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> Current StateTracker free memory was triggered by a timer. The default the
> timer value was DAGContext.STREAMING_WINDOW_SIZE_MILLIS.defaultValue *
> OperatorContext.APPLICATION_WINDOW_COUNT.defaultValue. It would have memory
> leak if the process with operator thread and memory release thread as
> following:
> bucket1: put(), put() ... put()
> bucket2: put(), put() ... put()
> freeMemory(): {bucket removed from bucketHeap and bucketAccessTimes}
> commit: bucket1, bucket2
> in this case, nothing was freed and the bucket can't be freed any more
> And the default value of free memory could large and memory used up even
> before get chance of free memory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
