[GitHub] wonook commented on a change in pull request #73: [NEMO-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [NEMO-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r203607677
##
File path: client/src/main/java/edu/snu/nemo/client/JobLauncher.java
##
@@ -109,36 +113,91 @@ public static void main(final String[] args) throws
Exception {
// Get DeployMode Conf
deployModeConf = Configurations.merge(getDeployModeConf(builtJobConf),
clientConf);
-// Launch client main
-runUserProgramMain(builtJobConf);
+// Start Driver and launch user program.
+try {
+ if (jobAndDriverConf == null || deployModeConf == null || builtJobConf
== null) {
Review comment:
It looks like that it could be that the methods assigning these variables
could be returning null. This code actually existed before the PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [NEMO-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [NEMO-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r203607354
##
File path: runtime/driver/src/main/java/edu/snu/nemo/driver/NemoDriver.java
##
@@ -155,14 +168,13 @@ public void onNext(final ActiveContext activeContext) {
}
/**
- * Start user application.
+ * Start user DAG.
*/
- public void startSchedulingUserApplication(final String dagString) {
-// Launch user application (with a new thread)
-final ExecutorService userApplicationRunnerThread =
Executors.newSingleThreadExecutor(
-new BasicThreadFactory.Builder().namingPattern("User App
thread-%d").build());
-userApplicationRunnerThread.execute(() ->
userApplicationRunner.run(dagString));
-userApplicationRunnerThread.shutdown();
+ public void startSchedulingUserDAG(final String dagString) {
+runnerThread.execute(() -> userApplicationRunner.run(dagString));
+// send driver notification that user application is done.
+runnerThread.execute(() ->
clientRPC.send(ControlMessage.DriverToClientMessage.newBuilder()
+
.setType(ControlMessage.DriverToClientMessageType.ExecutionDone).build()));
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [NEMO-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [NEMO-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r203607189
##
File path: client/src/main/java/edu/snu/nemo/client/JobLauncher.java
##
@@ -320,9 +381,13 @@ public static Configuration getBuiltJobConf() {
/**
* Get the collected data.
*
+ * @param the type of the data.
* @return the collected data.
*/
public static List getCollectedData() {
-return (List) collectedData;
+final List result = new ArrayList<>();
+result.addAll((List) COLLECTED_DATA);
Review comment:
Sure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [NEMO-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [NEMO-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r203606886
##
File path: client/src/main/java/edu/snu/nemo/client/JobLauncher.java
##
@@ -78,16 +84,14 @@ private JobLauncher() {
* @throws Exception exception on the way.
*/
public static void main(final String[] args) throws Exception {
-final DriverRPCServer driverRPCServer = new DriverRPCServer();
+driverRPCServer = new DriverRPCServer();
+
// Registers actions for launching the DAG.
driverRPCServer
.registerHandler(ControlMessage.DriverToClientMessageType.DriverStarted, event
-> { })
-
.registerHandler(ControlMessage.DriverToClientMessageType.ResourceReady, event
->
-
driverRPCServer.send(ControlMessage.ClientToDriverMessage.newBuilder()
- .setType(ControlMessage.ClientToDriverMessageType.LaunchDAG)
-
.setLaunchDAG(ControlMessage.LaunchDAGMessage.newBuilder().setDag(serializedDAG).build())
- .build()))
-
.registerHandler(ControlMessage.DriverToClientMessageType.DataCollected,
message -> collectedData.addAll(
+
.registerHandler(ControlMessage.DriverToClientMessageType.ResourceReady, event
-> driverReadyLatch.countDown())
Review comment:
I'll rename it to DriverReady. I tried to follow what was already there, but
I think it would be better to make the change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [NEMO-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [NEMO-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r203606454
##
File path: examples/spark/src/test/java/edu/snu/nemo/examples/spark/MRJava.java
##
@@ -66,8 +66,30 @@ public void testSparkWordCount() throws Exception {
ExampleTestUtil.deleteOutputFile(fileBasePath, outputFileName);
}
}
+ /* TODO #152: enable execution of multiple jobs (call scheduleJob multiple
times with caching).
+ @Test(timeout = TIMEOUT)
+ public void testSparkWordAndLineCount() throws Exception {
+final String inputFileName = "sample_input_wordcount_spark";
+final String outputFileName = "sample_output_word_and_line_count";
+final String testResourceFilename = "test_output_word_and_line_count";
+final String inputFilePath = fileBasePath + inputFileName;
+final String outputFilePath = fileBasePath + outputFileName;
+
+JobLauncher.main(builder
+.addJobId(JavaWordAndLineCount.class.getSimpleName() + "_test")
+.addUserMain(JavaWordAndLineCount.class.getCanonicalName())
+.addUserArgs(inputFilePath, outputFilePath)
+.addOptimizationPolicy(DefaultPolicy.class.getCanonicalName())
+.build());
+
+try {
+ ExampleTestUtil.ensureOutputValidity(fileBasePath, outputFileName,
testResourceFilename);
+} finally {
+ ExampleTestUtil.deleteOutputFile(fileBasePath, outputFileName);
+}
+ }
- /* Temporary disabled because of Travis issue
+ /* Temporary disabled due to Travis issue
Review comment:
Should this be reverted?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [NEMO-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [NEMO-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r203606399
##
File path:
common/src/main/java/edu/snu/nemo/common/ir/vertex/transform/Transform.java
##
@@ -56,7 +56,7 @@ default Object getTag() {
/**
* Context of the transform.
*/
- interface Context {
+ interface Context extends Serializable {
Review comment:
Do you think this change should be reverted?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [Nemo-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [Nemo-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r202942700
##
File path: client/src/main/java/edu/snu/nemo/client/JobLauncher.java
##
@@ -118,22 +153,39 @@ public static void main(final String[] args) throws
Exception {
*/
// When modifying the signature of this method, see
CompilerTestUtil#compileDAG and make corresponding changes
public static void launchDAG(final DAG dag) {
Review comment:
I'll add the comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [Nemo-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [Nemo-62] Support Multiple Jobs Submission in a Single User Program URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r202942040 ## File path: runtime/common/src/main/java/edu/snu/nemo/runtime/common/plan/PhysicalPlanGenerator.java ## @@ -79,7 +79,6 @@ private PhysicalPlanGenerator(final StagePartitioner stagePartitioner, // this is needed because of DuplicateEdgeGroupProperty. handleDuplicateEdgeGroupProperty(dagOfStages); - Review comment: It's a bit of a code cleanup This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [Nemo-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [Nemo-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r202282748
##
File path: examples/spark/src/test/java/edu/snu/nemo/examples/spark/MRJava.java
##
@@ -66,8 +66,30 @@ public void testSparkWordCount() throws Exception {
ExampleTestUtil.deleteOutputFile(fileBasePath, outputFileName);
}
}
+ /* TODO #152: enable execution of multiple jobs (call scheduleJob multiple
times with caching).
+ @Test(timeout = TIMEOUT)
+ public void testSparkWordAndLineCount() throws Exception {
+final String inputFileName = "sample_input_wordcount_spark";
+final String outputFileName = "sample_output_word_and_line_count";
+final String testResourceFilename = "test_output_word_and_line_count";
+final String inputFilePath = fileBasePath + inputFileName;
+final String outputFilePath = fileBasePath + outputFileName;
+
+JobLauncher.main(builder
+.addJobId(JavaWordAndLineCount.class.getSimpleName() + "_test")
+.addUserMain(JavaWordAndLineCount.class.getCanonicalName())
+.addUserArgs(inputFilePath, outputFilePath)
+.addOptimizationPolicy(DefaultPolicy.class.getCanonicalName())
+.build());
+
+try {
+ ExampleTestUtil.ensureOutputValidity(fileBasePath, outputFileName,
testResourceFilename);
+} finally {
+ ExampleTestUtil.deleteOutputFile(fileBasePath, outputFileName);
+}
+ }
Review comment:
It's actually intentional ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [Nemo-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [Nemo-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r202264147
##
File path: runtime/driver/src/main/java/edu/snu/nemo/driver/NemoDriver.java
##
@@ -96,8 +98,13 @@ private NemoDriver(final UserApplicationRunner
userApplicationRunner,
this.glusterDirectory = glusterDirectory;
this.handler = new RemoteClientMessageLoggingHandler(client);
this.clientRPC = clientRPC;
-
clientRPC.registerHandler(ControlMessage.ClientToDriverMessageType.LaunchDAG,
-message ->
startSchedulingUserApplication(message.getLaunchDAG().getDag()));
+
clientRPC.registerHandler(ControlMessage.ClientToDriverMessageType.LaunchDAG,
message ->
+startSchedulingUserApplication(message.getLaunchDAG().getDag()));
+
clientRPC.registerHandler(ControlMessage.ClientToDriverMessageType.DriverShutdown,
message -> {
+ LOG.info("Driver shutdown initiated");
+ userApplicationRunnerThread.execute(runtimeMaster::terminate);
+ userApplicationRunnerThread.shutdown();
Review comment:
Sure. I'll refactor it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [Nemo-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [Nemo-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r202262425
##
File path:
runtime/common/src/main/java/edu/snu/nemo/runtime/common/metric/StateTransitionEvent.java
##
@@ -15,11 +15,13 @@
*/
package edu.snu.nemo.runtime.common.metric;
+import java.io.Serializable;
+
/**
* Event of state transition. It contains timestamp and the state transition.
* @param class of state for the metric.
*/
-public final class StateTransitionEvent extends Event {
+public final class StateTransitionEvent extends Event {
Review comment:
It's to remove the SonarQube warning that was showing on my PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] wonook commented on a change in pull request #73: [Nemo-62] Support Multiple Jobs Submission in a Single User Program
wonook commented on a change in pull request #73: [Nemo-62] Support Multiple
Jobs Submission in a Single User Program
URL: https://github.com/apache/incubator-nemo/pull/73#discussion_r202254948
##
File path:
common/src/main/java/edu/snu/nemo/common/ir/vertex/transform/Transform.java
##
@@ -56,7 +56,7 @@ default Object getTag() {
/**
* Context of the transform.
*/
- interface Context {
+ interface Context extends Serializable {
Review comment:
It's to remove the SonarQube warning that was showing on my PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
