[
https://issues.apache.org/jira/browse/TAJO-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100332#comment-14100332
]
ASF GitHub Bot commented on TAJO-991:
-------------------------------------
Github user jinossy commented on a diff in the pull request:
https://github.com/apache/tajo/pull/107#discussion_r16338689
--- Diff:
tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
---
@@ -312,6 +377,63 @@ public ChannelPipeline getPipeline() throws Exception {
}
}
+
+ Map<String, ProcessingStatus> processingStatusMap = new
ConcurrentHashMap<String, ProcessingStatus>();
+
+ public void completeFileChunk(FadvisedFileRegion filePart,
+ String requestUri,
+ long startTime) {
+ ProcessingStatus status = processingStatusMap.get(requestUri);
+ if (status != null) {
+ status.decrementRemainFiles(filePart, startTime);
+ }
+ }
+
+ class ProcessingStatus {
+ String requestUri;
+ int numFiles;
+ AtomicInteger remainFiles;
+ long startTime;
+ long makeFileListTime;
+ long minTime = Long.MAX_VALUE;
+ long maxTime;
+ int numSlowFile;
+
+ public ProcessingStatus(String requestUri) {
+ this.requestUri = requestUri;
+ this.startTime = System.currentTimeMillis();
+ }
+
+ public void setNumFiles(int numFiles) {
+ this.numFiles = numFiles;
+ this.remainFiles = new AtomicInteger(numFiles);
+ }
+ public void decrementRemainFiles(FadvisedFileRegion filePart, long
fileStartTime) {
+ synchronized(remainFiles) {
+ long fileSendTime = System.currentTimeMillis() - fileStartTime;
+ if (fileSendTime > 20 * 1000) {
+ LOG.info("PullServer send too long time: filePos=" +
filePart.getPosition() + ", fileLen=" + filePart.getCount());
+ numSlowFile++;
+ }
+ if (fileSendTime > maxTime) {
+ maxTime = fileSendTime;
+ }
+ if (fileSendTime < minTime) {
+ minTime = fileSendTime;
+ }
+ int remain = remainFiles.decrementAndGet();
+ if (remain <= 0) {
+ synchronized(processingStatusMap) {
--- End diff --
remove this block
> Running PullServer on a dedicated JVM process which separates from worker.
> --------------------------------------------------------------------------
>
> Key: TAJO-991
> URL: https://issues.apache.org/jira/browse/TAJO-991
> Project: Tajo
> Issue Type: Sub-task
> Components: data shuffle
> Reporter: Hyoungjun Kim
> Assignee: Hyoungjun Kim
>
> Currently PullServer runs on TajoWorker. If a TajoWorker starts Garbage
> Collector, all fetching tasks which run on other worker are paused during
> Garbage Collection. Running PullServer on a dedicated JVM process which
> separates from worker is helpful avoiding stop the all cluster by GC.
> PullServer uses small memory and doesn't share any information and status
> with TajoWorker.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
