[jira] [Commented] (TEZ-3362) Delete intermediate data at DAG level for Shuffle Handler
[ https://issues.apache.org/jira/browse/TEZ-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503138#comment-17503138 ] Thomas Graves commented on TEZ-3362: [~jeagles] [~kshukla] I know its been a while, I was looking at this feature and I'm wondering how this works on a secure yarn setup? Generally the files and directories are read/write by the user and read only by the Hadoop group. If this runs in the auxiliary shuffle handler in the node manager it wouldn't have permissions to remove the directories. Is this somehow relying on other permission or configuration changes or is it running the remove as the user and I'm not seeing it? > Delete intermediate data at DAG level for Shuffle Handler > - > > Key: TEZ-3362 > URL: https://issues.apache.org/jira/browse/TEZ-3362 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Assignee: Kuhu Shukla >Priority: Major > Fix For: 0.9.0 > > Attachments: TEZ-3362.001.patch, TEZ-3362.002.patch, > TEZ-3362.003.patch, TEZ-3362.004.patch, TEZ-3362.005.patch, > TEZ-3362.006.patch, TEZ-3362.007.patch, TEZ-3362.008.patch > > > Applications like hive that use tez in session mode need the ability to > delete intermediate data after a DAG completes and while the application > continues to run. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [tez] shameersss1 commented on a change in pull request #60: TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler
shameersss1 commented on a change in pull request #60:
URL: https://github.com/apache/tez/pull/60#discussion_r821872406
##
File path:
tez-plugins/tez-aux-services/src/test/java/org/apache/tez/auxservices/TestShuffleHandler.java
##
@@ -1312,6 +1313,86 @@ protected void sendError(ChannelHandlerContext ctx,
String message,
}
}
+ @Test
+ public void testVertexDelete() throws Exception {
+final ArrayList failures = new ArrayList(1);
+Configuration conf = new Configuration();
+conf.setInt(ShuffleHandler.MAX_SHUFFLE_CONNECTIONS, 3);
+conf.setInt(ShuffleHandler.SHUFFLE_PORT_CONFIG_KEY, 0);
+conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
+"simple");
+UserGroupInformation.setConfiguration(conf);
+File absLogDir = new File("target", TestShuffleHandler.class.
+getSimpleName() + "LocDir").getAbsoluteFile();
+conf.set(YarnConfiguration.NM_LOCAL_DIRS, absLogDir.getAbsolutePath());
+ApplicationId appId = ApplicationId.newInstance(12345L, 1);
+String appAttemptId = "attempt_12345_0001_1_00_00_0_10003_0";
+String user = "randomUser";
+List fileMap = new ArrayList();
+String vertexDirStr =
+StringUtils.join(Path.SEPARATOR,
+new String[] { absLogDir.getAbsolutePath(),
+ShuffleHandler.USERCACHE, user,
+ShuffleHandler.APPCACHE, appId.toString(),
"dag_1/output/" + appAttemptId});
+File vertexDir = new File(vertexDirStr);
+Assert.assertFalse("Vetex Directory should not exists",
vertexDir.exists());
Review comment:
ack. I will fix this in next revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tez] shameersss1 commented on a change in pull request #60: TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler
shameersss1 commented on a change in pull request #60:
URL: https://github.com/apache/tez/pull/60#discussion_r821871999
##
File path:
tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java
##
@@ -1256,6 +1280,29 @@ private String getBaseLocation(String jobId, String
dagId, String user) {
return baseStr;
}
+/**
+ * Delete shuffle data in task directories belonging to a vertex.
+ */
+private void deleteTaskDirsOfVertex(String jobId, String dagId, String
vertexId, String user) throws IOException {
+ String baseStr = getBaseLocation(jobId, dagId, user);
+ FileContext lfc = FileContext.getLocalFSFileContext();
+ for(Path dagPath : lDirAlloc.getAllLocalPathsToRead(baseStr, conf)) {
+RemoteIterator status = lfc.listStatus(dagPath);
+final JobID jobID = JobID.forName(jobId);
+String taskDirPrefix = "attempt" + jobID.toString().replace("job", "")
+
+"_" + dagId + "_" + vertexId + "_";
+while (status.hasNext()) {
+ FileStatus fileStatus = status.next();
+ Path attemptPath = fileStatus.getPath();
+ if (attemptPath.getName().startsWith(taskDirPrefix)) {
+if(lfc.delete(attemptPath, true)) {
+ LOG.info("Deleted shuffle data in task directory : " +
attemptPath);
Review comment:
ack. I will fix this in next revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tez] shameersss1 commented on a change in pull request #60: TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler
shameersss1 commented on a change in pull request #60:
URL: https://github.com/apache/tez/pull/60#discussion_r821871579
##
File path:
tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java
##
@@ -1256,6 +1280,29 @@ private String getBaseLocation(String jobId, String
dagId, String user) {
return baseStr;
}
+/**
+ * Delete shuffle data in task directories belonging to a vertex.
+ */
+private void deleteTaskDirsOfVertex(String jobId, String dagId, String
vertexId, String user) throws IOException {
+ String baseStr = getBaseLocation(jobId, dagId, user);
+ FileContext lfc = FileContext.getLocalFSFileContext();
+ for(Path dagPath : lDirAlloc.getAllLocalPathsToRead(baseStr, conf)) {
+RemoteIterator status = lfc.listStatus(dagPath);
+final JobID jobID = JobID.forName(jobId);
+String taskDirPrefix = "attempt" + jobID.toString().replace("job", "")
+
Review comment:
ack. I will fix this in next revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tez] shameersss1 commented on a change in pull request #60: TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler
shameersss1 commented on a change in pull request #60:
URL: https://github.com/apache/tez/pull/60#discussion_r821870885
##
File path: tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/VertexImpl.java
##
@@ -3758,6 +3780,31 @@ public VertexState transition(VertexImpl vertex,
VertexEvent event) {
}
}
+ private static class VertexDeleteTransition implements
+ SingleArcTransition {
+
+@Override
+public void transition(VertexImpl vertex, VertexEvent event) {
+ vertex.incompleteChildrenVertices--;
+ // check if all the child vertices are completed
+ if (vertex.incompleteChildrenVertices == 0) {
+LOG.info("Vertex shuffle data deletion for vertex name: " +
+vertex.getName() + " with vertex id: " + vertex.getVertexId());
+// Get nodes of all the task attempts in vertex
+Set nodes = Sets.newHashSet();
+Map tasksMap = vertex.getTasks();
+tasksMap.keySet().forEach(taskId -> {
+ Map taskAttemptMap =
tasksMap.get(taskId).getAttempts();
+ taskAttemptMap.keySet().forEach(attemptId -> {
+nodes.add(taskAttemptMap.get(attemptId).getNodeId());
+ });
+});
+vertex.appContext.getAppMaster().vertexComplete(
+vertex.vertexId, nodes);
+ }
Review comment:
ack. I will fix this in next revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tez] shameersss1 commented on a change in pull request #60: TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler
shameersss1 commented on a change in pull request #60:
URL: https://github.com/apache/tez/pull/60#discussion_r821870419
##
File path: tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java
##
@@ -1772,6 +1774,52 @@ private static void parseVertexEdges(DAGImpl dag,
Map edgePlan
vertex.setInputVertices(inVertices);
vertex.setOutputVertices(outVertices);
+boolean cleanupShuffleDataAtVertexLevel =
dag.dagConf.getBoolean(TezConfiguration.TEZ_AM_VERTEX_CLEANUP_ON_COMPLETION,
+TezConfiguration.TEZ_AM_VERTEX_CLEANUP_ON_COMPLETION_DEFAULT) &&
ShuffleUtils.isTezShuffleHandler(dag.dagConf);
+if (cleanupShuffleDataAtVertexLevel) {
+ int deletionHeight =
dag.dagConf.getInt(TezConfiguration.TEZ_AM_VERTEX_CLEANUP_HEIGHT,
+ TezConfiguration.TEZ_AM_VERTEX_CLEANUP_HEIGHT_DEFAULT);
+ getSpannedVerticesAncestors(vertex, ancestors, deletionHeight);
Review comment:
ack. I will fix this in next revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tez] shameersss1 commented on a change in pull request #60: TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler
shameersss1 commented on a change in pull request #60:
URL: https://github.com/apache/tez/pull/60#discussion_r821868285
##
File path: tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
##
@@ -883,6 +883,25 @@ public TezConfiguration(boolean loadDefaults) {
+ "dag.cleanup.on.completion";
public static final boolean TEZ_AM_DAG_CLEANUP_ON_COMPLETION_DEFAULT = false;
+ /**
+ * Boolean value. Instructs AM to delete vertex shuffle data if a vertex and
all its
+ * child vertices at a certain depth are completed.
+ */
+ @ConfigurationScope(Scope.AM)
+ @ConfigurationProperty(type="boolean")
+ public static final String TEZ_AM_VERTEX_CLEANUP_ON_COMPLETION =
TEZ_AM_PREFIX
+ + "vertex.cleanup.on.completion";
+ public static final boolean TEZ_AM_VERTEX_CLEANUP_ON_COMPLETION_DEFAULT =
false;
+
+ /**
+ * Int value. The height from the vertex that it can issue shuffle data
deletion upon completion
+ */
+ @ConfigurationScope(Scope.AM)
+ @ConfigurationProperty(type="integer")
+ public static final String TEZ_AM_VERTEX_CLEANUP_HEIGHT = TEZ_AM_PREFIX
+ + "vertex.cleanup.height";
+ public static final int TEZ_AM_VERTEX_CLEANUP_HEIGHT_DEFAULT = 1;
Review comment:
Just to add to my previous comment, The intention behind having height
as a config is that (rather than value being 1 is that), The probability of
shuffle data of great ancestor(s) being requested is very rare and happens when
multiple shuffle nodes are lost. So having control over which ancestor(s)
shuffle data to clear up will give user more flexibility to control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tez] shameersss1 commented on a change in pull request #60: TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler
shameersss1 commented on a change in pull request #60:
URL: https://github.com/apache/tez/pull/60#discussion_r821865132
##
File path: tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
##
@@ -883,6 +883,25 @@ public TezConfiguration(boolean loadDefaults) {
+ "dag.cleanup.on.completion";
public static final boolean TEZ_AM_DAG_CLEANUP_ON_COMPLETION_DEFAULT = false;
+ /**
+ * Boolean value. Instructs AM to delete vertex shuffle data if a vertex and
all its
+ * child vertices at a certain depth are completed.
+ */
+ @ConfigurationScope(Scope.AM)
Review comment:
ack. I will fix this in next revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tez] tez-yetus commented on pull request #190: TEZ-4383: upgrade to mockito 4.3.1
tez-yetus commented on pull request #190: URL: https://github.com/apache/tez/pull/190#issuecomment-1061738481 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 19m 18s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 61 new or modified test files. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 5m 48s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 9m 55s | master passed | | +1 :green_heart: | compile | 10m 1s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | compile | 9m 52s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | checkstyle | 6m 13s | master passed | | +1 :green_heart: | javadoc | 8m 23s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 7m 10s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +0 :ok: | spotbugs | 0m 41s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +0 :ok: | findbugs | 6m 25s | root in master has 2 extant findbugs warnings. | | +0 :ok: | findbugs | 0m 39s | tez-tools/analyzers/job-analyzer in master has 4 extant findbugs warnings. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | -1 :x: | mvninstall | 0m 20s | tez-runtime-library in the patch failed. | | -1 :x: | mvninstall | 0m 18s | tez-protobuf-history-plugin in the patch failed. | | +1 :green_heart: | compile | 10m 3s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javac | 10m 3s | the patch passed | | +1 :green_heart: | compile | 8m 55s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 8m 55s | the patch passed | | +1 :green_heart: | checkstyle | 0m 23s | The patch passed checkstyle in tez-api | | +1 :green_heart: | checkstyle | 0m 17s | The patch passed checkstyle in tez-common | | +1 :green_heart: | checkstyle | 0m 18s | The patch passed checkstyle in tez-runtime-internals | | +1 :green_heart: | checkstyle | 0m 31s | tez-runtime-library: The patch generated 0 new + 310 unchanged - 23 fixed = 310 total (was 333) | | +1 :green_heart: | checkstyle | 0m 23s | The patch passed checkstyle in tez-mapreduce | | +1 :green_heart: | checkstyle | 0m 51s | tez-dag: The patch generated 0 new + 673 unchanged - 7 fixed = 673 total (was 680) | | +1 :green_heart: | checkstyle | 0m 19s | The patch passed checkstyle in tez-ext-service-tests | | +1 :green_heart: | checkstyle | 0m 16s | The patch passed checkstyle in tez-protobuf-history-plugin | | +1 :green_heart: | checkstyle | 0m 29s | tez-plugins/tez-yarn-timeline-history: The patch generated 0 new + 38 unchanged - 19 fixed = 38 total (was 57) | | +1 :green_heart: | checkstyle | 0m 17s | The patch passed checkstyle in tez-yarn-timeline-history-with-acls | | +1 :green_heart: | checkstyle | 0m 18s | tez-plugins/tez-yarn-timeline-history-with-fs: The patch generated 0 new + 22 unchanged - 3 fixed = 22 total (was 25) | | +1 :green_heart: | checkstyle | 0m 49s | The patch passed checkstyle in tez-history-parser | | +1 :green_heart: | checkstyle | 0m 38s | The patch passed checkstyle in tez-aux-services | | +1 :green_heart: | checkstyle | 0m 16s | The patch passed checkstyle in job-analyzer | | +1 :green_heart: | checkstyle | 1m 7s | root: The patch generated 0 new + 1193 unchanged - 52 fixed = 1193 total (was 1245) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 21s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 8m 11s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 | | +1 :green_heart: | javadoc | 7m 16s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | findbugs | 20m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 15s | tez-api in the patch passed. | | +1 :green_heart: | unit | 0m 33s | tez-common in the patch passed. | | +1 :green_heart: | unit | 0m 45s | tez-runtime-internals in the patch passed. | | +1 :green_heart: | unit | 4m 49s |
[GitHub] [tez] shameersss1 commented on a change in pull request #60: TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler
shameersss1 commented on a change in pull request #60:
URL: https://github.com/apache/tez/pull/60#discussion_r821509356
##
File path: tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
##
@@ -883,6 +883,25 @@ public TezConfiguration(boolean loadDefaults) {
+ "dag.cleanup.on.completion";
public static final boolean TEZ_AM_DAG_CLEANUP_ON_COMPLETION_DEFAULT = false;
+ /**
+ * Boolean value. Instructs AM to delete vertex shuffle data if a vertex and
all its
+ * child vertices at a certain depth are completed.
+ */
+ @ConfigurationScope(Scope.AM)
+ @ConfigurationProperty(type="boolean")
+ public static final String TEZ_AM_VERTEX_CLEANUP_ON_COMPLETION =
TEZ_AM_PREFIX
+ + "vertex.cleanup.on.completion";
+ public static final boolean TEZ_AM_VERTEX_CLEANUP_ON_COMPLETION_DEFAULT =
false;
+
+ /**
+ * Int value. The height from the vertex that it can issue shuffle data
deletion upon completion
+ */
+ @ConfigurationScope(Scope.AM)
+ @ConfigurationProperty(type="integer")
+ public static final String TEZ_AM_VERTEX_CLEANUP_HEIGHT = TEZ_AM_PREFIX
+ + "vertex.cleanup.height";
+ public static final int TEZ_AM_VERTEX_CLEANUP_HEIGHT_DEFAULT = 1;
Review comment:
Then all the vertex shuffle data of Map 1 will be deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [tez] shameersss1 commented on a change in pull request #60: TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler
shameersss1 commented on a change in pull request #60:
URL: https://github.com/apache/tez/pull/60#discussion_r821509356
##
File path: tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java
##
@@ -883,6 +883,25 @@ public TezConfiguration(boolean loadDefaults) {
+ "dag.cleanup.on.completion";
public static final boolean TEZ_AM_DAG_CLEANUP_ON_COMPLETION_DEFAULT = false;
+ /**
+ * Boolean value. Instructs AM to delete vertex shuffle data if a vertex and
all its
+ * child vertices at a certain depth are completed.
+ */
+ @ConfigurationScope(Scope.AM)
+ @ConfigurationProperty(type="boolean")
+ public static final String TEZ_AM_VERTEX_CLEANUP_ON_COMPLETION =
TEZ_AM_PREFIX
+ + "vertex.cleanup.on.completion";
+ public static final boolean TEZ_AM_VERTEX_CLEANUP_ON_COMPLETION_DEFAULT =
false;
+
+ /**
+ * Int value. The height from the vertex that it can issue shuffle data
deletion upon completion
+ */
+ @ConfigurationScope(Scope.AM)
+ @ConfigurationProperty(type="integer")
+ public static final String TEZ_AM_VERTEX_CLEANUP_HEIGHT = TEZ_AM_PREFIX
+ + "vertex.cleanup.height";
+ public static final int TEZ_AM_VERTEX_CLEANUP_HEIGHT_DEFAULT = 1;
Review comment:
Then all the vextex shuffle data of Map 1 will be deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
