[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332969&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332969 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 23/Oct/19 23:37 Start Date: 23/Oct/19 23:37 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332969) Time Spent: 2h 20m (was: 2h 10m) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332242&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332242 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 21:37 Start Date: 22/Oct/19 21:37 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#issuecomment-540841640 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=h1) Report > Merging [#2760](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/28d3d7b5b07b82d86452f0e423559f104a383152?src=pr&el=desc) will **increase** coverage by `41.2%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2760 +/- ## + Coverage 4.15% 45.35% +41.2% - Complexity 744 8861+8117 Files 1894 1894 Lines 7082970829 Branches 7787 7787 + Hits 294232125 +29183 + Misses6757435743 -31831 - Partials313 2961+2648 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh) | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: | | [...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh) | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: | | [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/incubator-gobbl
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=33&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-33 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 21:01 Start Date: 22/Oct/19 21:01 Worklog Time Spent: 10m Work Description: Will-Lo commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337748160 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/application.yaml ## @@ -0,0 +1,80 @@ +# In the future, build the kubernetes cluster from the official docker account +# Also ensure that proper tagging/versioning is done i.e. remove :latest tag and instead use the digest of the container + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: 'gaas-app' + labels: +app: gaas-app +spec: + selector: +matchLabels: + app: gaas-app + replicas: 1 + template: +metadata: + name: 'gaas-app' + labels: +app: gaas-app +spec: + volumes: +- name: 'shared-jobs' + persistentVolumeClaim: +claimName: shared-jobs-claim +- name: 'shared-template-catalogs' + persistentVolumeClaim: +claimName: shared-template-catalogs-claim + containers: +- name: gobblin-service + image: will97/gobblin-as-a-service:latest + volumeMounts: +- name: shared-jobs + mountPath: /tmp/gobblin-service/jobs +- name: shared-template-catalogs + mountPath: /tmp/templateCatalog + +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: 'gobblin-standalone-app' + labels: +app: gobblin-standalone-app +spec: + selector: +matchLabels: + app: gobblin-standalone-app + replicas: 1 + template: +metadata: + name: 'gobblin-standalone-app' + labels: +app: gobblin-standalone-app +spec: + volumes: +- name: 'shared-jobs' + persistentVolumeClaim: +claimName: shared-jobs-claim + containers: + +- name: gobblin-standalone + image: will97/gobblin-standalone:latest + volumeMounts: +- name: shared-jobs + mountPath: /tmp/gobblin-standalone/jobs +--- +apiVersion: v1 +kind: Service +metadata: + name: gaas-svc + labels: +app: gobblin-service +spec: + type: NodePort Review comment: Okay I'll change the service type to a clusterIP This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 33) Time Spent: 2h (was: 1h 50m) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332127&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332127 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 18:14 Start Date: 22/Oct/19 18:14 Worklog Time Spent: 10m Work Description: Will-Lo commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337673408 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/application.yaml ## @@ -0,0 +1,80 @@ +# In the future, build the kubernetes cluster from the official docker account +# Also ensure that proper tagging/versioning is done i.e. remove :latest tag and instead use the digest of the container + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: 'gaas-app' + labels: +app: gaas-app +spec: + selector: +matchLabels: + app: gaas-app + replicas: 1 + template: +metadata: + name: 'gaas-app' Review comment: Theres no collision because in kubernetes they differentiate objects of different types. Though I agree its going to get confusing so I'll rename the deployment to `gaas-deployment` and `gobblin-standalone-deployment`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332127) Time Spent: 1h 50m (was: 1h 40m) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332124&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332124 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 17:58 Start Date: 22/Oct/19 17:58 Worklog Time Spent: 10m Work Description: Will-Lo commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337665813 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/ingress.yaml ## @@ -0,0 +1,15 @@ +apiVersion: networking.k8s.io/v1beta1 +kind: Ingress +metadata: + name: test-ingress + annotations: +nginx.ingress.kubernetes.io/rewrite-target: /$1 +spec: + rules: +- host: gaas-test.info Review comment: I would say it's not needed for a minimal ingress as the URL mapping is done on my local machine's side. I can remove this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332124) Time Spent: 1h 40m (was: 1.5h) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332119&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332119 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 17:52 Start Date: 22/Oct/19 17:52 Worklog Time Spent: 10m Work Description: Will-Lo commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337663054 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/ingress.yaml ## @@ -0,0 +1,15 @@ +apiVersion: networking.k8s.io/v1beta1 +kind: Ingress +metadata: + name: test-ingress + annotations: +nginx.ingress.kubernetes.io/rewrite-target: /$1 Review comment: I don't think so, I'll clean this up, thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332119) Time Spent: 1.5h (was: 1h 20m) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332118&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332118 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 17:52 Start Date: 22/Oct/19 17:52 Worklog Time Spent: 10m Work Description: Will-Lo commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337662901 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/storage.yaml ## @@ -0,0 +1,54 @@ +apiVersion: v1 +kind: PersistentVolume +metadata: + name: shared-jobs-volume +spec: + capacity: +storage: 100Mi + volumeMode: Filesystem + accessModes: +- ReadWriteOnce + persistentVolumeReclaimPolicy: Delete + storageClassName: manual + hostPath: +path: "/etc/opt/job-conf" +--- +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: shared-jobs-claim +spec: + accessModes: +- ReadWriteOnce + storageClassName: manual + resources: +requests: + storage: 100Mi +--- +apiVersion: v1 +kind: PersistentVolume Review comment: Each definition describes resources available to the cluster, so yeah they're all their own instances This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332118) Time Spent: 1h 20m (was: 1h 10m) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332079&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332079 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 16:40 Start Date: 22/Oct/19 16:40 Worklog Time Spent: 10m Work Description: zxcware commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337126532 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/storage.yaml ## @@ -0,0 +1,54 @@ +apiVersion: v1 +kind: PersistentVolume +metadata: + name: shared-jobs-volume +spec: + capacity: +storage: 100Mi + volumeMode: Filesystem + accessModes: +- ReadWriteOnce + persistentVolumeReclaimPolicy: Delete + storageClassName: manual + hostPath: +path: "/etc/opt/job-conf" +--- +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: shared-jobs-claim +spec: + accessModes: +- ReadWriteOnce + storageClassName: manual + resources: +requests: + storage: 100Mi +--- +apiVersion: v1 +kind: PersistentVolume Review comment: So, one `PV` definition in yaml represents one instance? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332079) Time Spent: 50m (was: 40m) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332077&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332077 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 16:40 Start Date: 22/Oct/19 16:40 Worklog Time Spent: 10m Work Description: zxcware commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337626492 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/ingress.yaml ## @@ -0,0 +1,15 @@ +apiVersion: networking.k8s.io/v1beta1 +kind: Ingress +metadata: + name: test-ingress + annotations: +nginx.ingress.kubernetes.io/rewrite-target: /$1 +spec: + rules: +- host: gaas-test.info Review comment: Is this needed? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332077) Time Spent: 0.5h (was: 20m) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332081&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332081 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 16:40 Start Date: 22/Oct/19 16:40 Worklog Time Spent: 10m Work Description: zxcware commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337627624 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/ingress.yaml ## @@ -0,0 +1,15 @@ +apiVersion: networking.k8s.io/v1beta1 +kind: Ingress +metadata: + name: test-ingress + annotations: +nginx.ingress.kubernetes.io/rewrite-target: /$1 Review comment: Is `rewritten` needed as we still keep the original url here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332081) Time Spent: 1h 10m (was: 1h) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332078&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332078 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 16:40 Start Date: 22/Oct/19 16:40 Worklog Time Spent: 10m Work Description: zxcware commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337098085 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/application.yaml ## @@ -0,0 +1,80 @@ +# In the future, build the kubernetes cluster from the official docker account +# Also ensure that proper tagging/versioning is done i.e. remove :latest tag and instead use the digest of the container + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: 'gaas-app' + labels: +app: gaas-app +spec: + selector: +matchLabels: + app: gaas-app + replicas: 1 + template: +metadata: + name: 'gaas-app' Review comment: Each object should have a unique name. Does template name collide with the deployment name here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332078) Time Spent: 40m (was: 0.5h) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=332080&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-332080 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 22/Oct/19 16:40 Start Date: 22/Oct/19 16:40 Worklog Time Spent: 10m Work Description: zxcware commented on pull request #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#discussion_r337622515 ## File path: gobblin-kubernetes/gobblin-service/basic-cluster/application.yaml ## @@ -0,0 +1,80 @@ +# In the future, build the kubernetes cluster from the official docker account +# Also ensure that proper tagging/versioning is done i.e. remove :latest tag and instead use the digest of the container + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: 'gaas-app' + labels: +app: gaas-app +spec: + selector: +matchLabels: + app: gaas-app + replicas: 1 + template: +metadata: + name: 'gaas-app' + labels: +app: gaas-app +spec: + volumes: +- name: 'shared-jobs' + persistentVolumeClaim: +claimName: shared-jobs-claim +- name: 'shared-template-catalogs' + persistentVolumeClaim: +claimName: shared-template-catalogs-claim + containers: +- name: gobblin-service + image: will97/gobblin-as-a-service:latest + volumeMounts: +- name: shared-jobs + mountPath: /tmp/gobblin-service/jobs +- name: shared-template-catalogs + mountPath: /tmp/templateCatalog + +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: 'gobblin-standalone-app' + labels: +app: gobblin-standalone-app +spec: + selector: +matchLabels: + app: gobblin-standalone-app + replicas: 1 + template: +metadata: + name: 'gobblin-standalone-app' + labels: +app: gobblin-standalone-app +spec: + volumes: +- name: 'shared-jobs' + persistentVolumeClaim: +claimName: shared-jobs-claim + containers: + +- name: gobblin-standalone + image: will97/gobblin-standalone:latest + volumeMounts: +- name: shared-jobs + mountPath: /tmp/gobblin-standalone/jobs +--- +apiVersion: v1 +kind: Service +metadata: + name: gaas-svc + labels: +app: gobblin-service +spec: + type: NodePort Review comment: `Ingress` and `NodePort` are 2 different way to expose a Service to the outside. If we have one, we don't need the other. Checkout https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types (search for string "You can also use Ingress to expose your Service") This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 332080) Time Spent: 1h (was: 50m) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Defines a kubernetes cluster to demonstrate a workflow between GaaS and > Gobblin Standalone, all living in a K8s environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[ https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=326635&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326635 ] ASF GitHub Bot logged work on GOBBLIN-906: -- Author: ASF GitHub Bot Created on: 10/Oct/19 23:39 Start Date: 10/Oct/19 23:39 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2760: [GOBBLIN-906] Initializes kubernetes cluster for GaaS and Gobblin Standalone URL: https://github.com/apache/incubator-gobblin/pull/2760#issuecomment-540841640 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=h1) Report > Merging [#2760](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/928e0180c471fc4b7a6caee041b001b5b34e1cc6?src=pr&el=desc) will **increase** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2760 +/- ## + Coverage 45.27% 45.29% +0.01% - Complexity 8830 8834 +4 Files 1889 1889 Lines 7064270642 Branches 7759 7759 + Hits 3198531994 +9 + Misses3570335698 -5 + Partials 2954 2950 -4 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `78.12% <0%> (-1.57%)` | `15% <0%> (-1%)` | | | [...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==) | `30.87% <0%> (+0.67%)` | `25% <0%> (+1%)` | :arrow_up: | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `64.44% <0%> (+1.11%)` | `16% <0%> (+1%)` | :arrow_up: | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `65.27% <0%> (+1.38%)` | `28% <0%> (ø)` | :arrow_down: | | [...ava/org/apache/gobblin/fsm/FiniteStateMachine.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2760/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2ZzbS9GaW5pdGVTdGF0ZU1hY2hpbmUuamF2YQ==) | `76.51% <0%> (+3.03%)` | `21% <0%> (+3%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=footer). Last update [928e018...dbf201b](https://codecov.io/gh/apache/incubator-gobblin/pull/2760?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 326635) Time Spent: 20m (was: 10m) > Define Kubernetes Cluster for GaaS and Gobblin Standalone > - > > Key: GOBBLIN-906 > URL: https://issues.apache.org/jira/browse/GOBBLIN-906 > Project: Apache Gobblin > Issue Type: New Feature >Affects Versions:
[jira] [Work logged] (GOBBLIN-906) Define Kubernetes Cluster for GaaS and Gobblin Standalone
[
https://issues.apache.org/jira/browse/GOBBLIN-906?focusedWorklogId=326604&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-326604
]
ASF GitHub Bot logged work on GOBBLIN-906:
--
Author: ASF GitHub Bot
Created on: 10/Oct/19 22:12
Start Date: 10/Oct/19 22:12
Worklog Time Spent: 10m
Work Description: Will-Lo commented on pull request #2760: [GOBBLIN-906]
Initializes kubernetes cluster for GaaS and Gobblin Standalone
URL: https://github.com/apache/incubator-gobblin/pull/2760
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-906
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
Adds yaml files to define kubernetes deployment for GaaS and Gobblin
Standalone
Utilizes the node's local filesystem as the communication mechanism between
GaaS and Gobblin Standalone.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Testing the application on Kubernetes
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 326604)
Remaining Estimate: 0h
Time Spent: 10m
> Define Kubernetes Cluster for GaaS and Gobblin Standalone
> -
>
> Key: GOBBLIN-906
> URL: https://issues.apache.org/jira/browse/GOBBLIN-906
> Project: Apache Gobblin
> Issue Type: New Feature
>Affects Versions: 0.15.0
>Reporter: William Lo
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Defines a kubernetes cluster to demonstrate a workflow between GaaS and
> Gobblin Standalone, all living in a K8s environment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
