autumnust commented on a change in pull request #3176:
URL: https://github.com/apache/incubator-gobblin/pull/3176#discussion_r547453005
##########
File path:
gobblin-modules/gobblin-azkaban/src/test/java/org/apache/gobblin/service/modules/orchestration/AzkabanClientTest.java
##########
@@ -65,55 +63,73 @@ public void setup() throws Exception {
.build();
}
+ @BeforeMethod
+ public void testSetup() {
+ projectName = "test-project-" + System.currentTimeMillis() + "-" +
UUID.randomUUID().toString().substring(0, 4);
+ description = "This is test project.";
+ }
+
+ @AfterMethod
+ public void testCleanup() throws AzkabanClientException {
+ this.client.deleteProject(projectName);
+ }
+
@AfterClass
public void cleanup() throws IOException {
this.client.close();
}
private void ensureProjectExist(String projectName, String description)
throws AzkabanClientException {
- // make sure it is in a clean state
- this.client.deleteProject(projectName);
-
- // make sure the project is created successfully
this.client.createProject(projectName, description);
}
- @Test(enabled = false)
- public void testFetchLog() throws AzkabanClientException {
- String execId = "11211956";
- String jobId = "tracking-hourly-bucket1";
-
- // fetch log
- this.client.fetchExecutionLog(execId, jobId, "0", "100000000", new
File("/tmp/sample.log"));
- }
+ public void testFetchLog() throws Exception {
+ String flowName = "test-exec-flow";
+ String jobId = "test-exec-flow";
+ ensureProjectExist(projectName, description);
+ File zipFile = createAzkabanZip(flowName);
+ this.client.uploadProjectZip(projectName, zipFile);
- @Test(enabled = false)
- public void testCreateProject() throws AzkabanClientException {
- String projectName = "project-create";
- String description = "This is a create project test.";
+ AzkabanExecuteFlowStatus execStatus = this.client.executeFlow(projectName,
flowName, Maps.newHashMap());
+ String execId = execStatus.getResponse().getExecId();
+
+ ByteArrayOutputStream logStream = null;
+ int maxTries = 10;
Review comment:
Considering smaller value?
##########
File path:
gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/service/modules/orchestration/AzkabanClient.java
##########
@@ -97,7 +86,8 @@ protected AzkabanClient(String username,
long sessionExpireInMin,
CloseableHttpClient httpClient,
SessionManager sessionManager,
- ExecutorService executorService)
+ ExecutorService executorService,
Review comment:
Since it is a protected method, shall we maintain the backward
compatibility ?
##########
File path:
gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/service/modules/orchestration/AzkabanProjectFlowsStatus.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.service.modules.orchestration;
+
+import lombok.AllArgsConstructor;
+import lombok.Getter;
+
+import java.util.List;
+
+public class AzkabanProjectFlowsStatus extends
AzkabanClientStatus<AzkabanProjectFlowsStatus.Project> {
Review comment:
I am not quite get the need for this class with two static inner class
within it. Can you add a javadoc, and why this is needed ( Can the `Project`
and `Flow` class themselves alone serve the same purpose? )
##########
File path:
gobblin-modules/gobblin-azkaban/src/test/java/org/apache/gobblin/service/modules/orchestration/AzkabanClientTest.java
##########
@@ -48,9 +41,14 @@
* Please check https://azkaban.github.io/azkaban/docs/latest/ for how to
setup Azkaban-solo-server.
*/
@Slf4j
+@Test(enabled = false)
Review comment:
Intentional ?
##########
File path:
gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/service/modules/orchestration/AzkabanClient.java
##########
@@ -306,6 +317,22 @@ public AzkabanClientStatus deleteProject(String
projectName) throws AzkabanClien
return runWithRetry(callable, AzkabanClientStatus.class);
}
+ /**
+ * Checks if the project with specified name exists in Azkaban
+ */
+ public Boolean projectExists(String projectName) throws
AzkabanClientException {
+ try {
+ fetchProjectFlows(projectName);
+ return true;
+ } catch (AzkabanClientException e) {
+ // Azkaban does not return a strongly typed error code, so we are
checking the message
+ if (e.getCause().getMessage().contains("doesn't exist")) {
+ return false;
Review comment:
Shall we print the error message anyway ? relying on a strict string
matching here seems to be not reliable enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
