Re: Gobblin on Yarn ?

Hi Sudarshan MR mode, will have dependency on hadoop cluster, I am thinking to have independent gobblin cluster for all the data movement jobs. also I have tried Hive-Distcp on cluster mode and managed to run it. ( there are lot

Job Management

Hi, I see gobblin creates jobs and tasks and put into zookeeper for worker nodes to pickup and process. ( not sure if this only applies to cluster mode or all modes ) In case of job failures, how one suppose to restart, ignore or skip the rest of the job or even disable it? is there an API for

writer and publisher

Hi Sudarshan, had couple of questions, thought you might help me figure it out. 1. How gobblin can integrate with schema registry and how it support schema evolution ? 2. based on class info, is there a way to remove or mask the column ? 3. how about encryption, is that something, connector has

Gobblin at ApacheCon ?

Hi Guys, Lets present Apache Gobblin at the ApacheCon. I would be interested in presenting/co-presenting PayPal's use-case. @PMCs, Please share your thoughts. Thanks Jay

[jira] [Work logged] (GOBBLIN-712) Add version strategy for configbased dataset copy

[ https://issues.apache.org/jira/browse/GOBBLIN-712?focusedWorklogId=222431=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222431 ] ASF GitHub Bot logged work on GOBBLIN-712: -- Author: ASF GitHub Bot

[GitHub] [incubator-gobblin] asfgit closed pull request #2584: Filter Out Empty MultiWorkUnits

asfgit closed pull request #2584: Filter Out Empty MultiWorkUnits URL: https://github.com/apache/incubator-gobblin/pull/2584 This is an automated message from the Apache Git Service. To respond to the message, please log on

[jira] [Resolved] (GOBBLIN-717) Filter Out Empty MultiWorkUnits

[ https://issues.apache.org/jira/browse/GOBBLIN-717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-717. --- Resolution: Fixed Fix Version/s: 0.15.0 Issue resolved by pull request #2584

[jira] [Work logged] (GOBBLIN-707) combine & standardize all gobblin scripts into one master script & restructure configs accordingly

[ https://issues.apache.org/jira/browse/GOBBLIN-707?focusedWorklogId=222609=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222609 ] ASF GitHub Bot logged work on GOBBLIN-707: -- Author: ASF GitHub Bot

[jira] [Work logged] (GOBBLIN-707) combine & standardize all gobblin scripts into one master script & restructure configs accordingly

[ https://issues.apache.org/jira/browse/GOBBLIN-707?focusedWorklogId=222610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222610 ] ASF GitHub Bot logged work on GOBBLIN-707: -- Author: ASF GitHub Bot

[jira] [Work logged] (GOBBLIN-707) combine & standardize all gobblin scripts into one master script & restructure configs accordingly

[ https://issues.apache.org/jira/browse/GOBBLIN-707?focusedWorklogId=222611=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222611 ] ASF GitHub Bot logged work on GOBBLIN-707: -- Author: ASF GitHub Bot

[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command

autumnust commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command URL: https://github.com/apache/incubator-gobblin/pull/2578#discussion_r271922811 ## File path: conf/yarn/reference.conf ## @@ -38,6 +38,6 @@

[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command

autumnust commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command URL: https://github.com/apache/incubator-gobblin/pull/2578#discussion_r271925108 ## File path: gobblin-docs/user-guide/Gobblin-CLI.md ## @@ -28,29

[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command

autumnust commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command URL: https://github.com/apache/incubator-gobblin/pull/2578#discussion_r271923038 ## File path: conf/yarn/application.conf ## @@ -22,15 +22,18 @@

Re: writer and publisher

Hi Jay, 1. We have KafkaSchemaRegistry interface built in with several implementation. The schema was fetched where kafka byte being decoded, either in Extractor or Converter. There's no handling of schema evolution on Gobblin runtime ( @Sudarshan Vasudevan

Re: writer and publisher

Hi Jay, 1) Checkout `org.apache.gobblin.metrics.kafka.KafkaSchemaRegistry`. We're following the kafka way to support schema evolution. You may find more details here: https://docs.confluent.io/current/schema-registry/index.html 2) Were you asking for a converter that removes fields from a

[GitHub] [incubator-gobblin] ibuenros commented on issue #2579: [GOBBLIN-712] Add version strategy pickup for ConfigBasedDataset distcp workflow

ibuenros commented on issue #2579: [GOBBLIN-712] Add version strategy pickup for ConfigBasedDataset distcp workflow URL: https://github.com/apache/incubator-gobblin/pull/2579#issuecomment-479582467 +1 This is an automated

[jira] [Work logged] (GOBBLIN-719) gobblin-docs has invalid git links

[ https://issues.apache.org/jira/browse/GOBBLIN-719?focusedWorklogId=222440=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222440 ] ASF GitHub Bot logged work on GOBBLIN-719: -- Author: ASF GitHub Bot

[GitHub] [incubator-gobblin] yukuai518 commented on issue #2586: [GOBBLIN-719] fix invalid git links for classes in docs

yukuai518 commented on issue #2586: [GOBBLIN-719] fix invalid git links for classes in docs URL: https://github.com/apache/incubator-gobblin/pull/2586#issuecomment-479586004 +1 Thanks for fixing all the link issues!

[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #2589: [GOBBLIN-722] Unschedule gaas flow

arjun4084346 commented on a change in pull request #2589: [GOBBLIN-722] Unschedule gaas flow URL: https://github.com/apache/incubator-gobblin/pull/2589#discussion_r272010852 ## File path: gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/JobScheduler.java ## @@

[GitHub] [incubator-gobblin] arjun4084346 opened a new pull request #2589: [GOBBLIN-722] Unschedule gaas flow

arjun4084346 opened a new pull request #2589: [GOBBLIN-722] Unschedule gaas flow URL: https://github.com/apache/incubator-gobblin/pull/2589 Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

[jira] [Work logged] (GOBBLIN-722) add option to unschedule a gaas flow

[ https://issues.apache.org/jira/browse/GOBBLIN-722?focusedWorklogId=222746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222746 ] ASF GitHub Bot logged work on GOBBLIN-722: -- Author: ASF GitHub Bot

[jira] [Resolved] (GOBBLIN-712) Add version strategy for configbased dataset copy

[ https://issues.apache.org/jira/browse/GOBBLIN-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-712. --- Resolution: Fixed Fix Version/s: 0.15.0 Issue resolved by pull request #2579

[jira] [Created] (GOBBLIN-722) add option to unschedule a gaas flow

Arjun Singh Bora created GOBBLIN-722: Summary: add option to unschedule a gaas flow Key: GOBBLIN-722 URL: https://issues.apache.org/jira/browse/GOBBLIN-722 Project: Apache Gobblin Issue

[jira] [Work logged] (GOBBLIN-712) Add version strategy for configbased dataset copy

[ https://issues.apache.org/jira/browse/GOBBLIN-712?focusedWorklogId=222737=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222737 ] ASF GitHub Bot logged work on GOBBLIN-712: -- Author: ASF GitHub Bot

[GitHub] [incubator-gobblin] asfgit closed pull request #2579: [GOBBLIN-712] Add version strategy pickup for ConfigBasedDataset distcp workflow

asfgit closed pull request #2579: [GOBBLIN-712] Add version strategy pickup for ConfigBasedDataset distcp workflow URL: https://github.com/apache/incubator-gobblin/pull/2579 This is an automated message from the Apache Git

[jira] [Work logged] (GOBBLIN-722) add option to unschedule a gaas flow

[ https://issues.apache.org/jira/browse/GOBBLIN-722?focusedWorklogId=222796=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222796 ] ASF GitHub Bot logged work on GOBBLIN-722: -- Author: ASF GitHub Bot

[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2589: [GOBBLIN-722] Unschedule gaas flow

sv2000 commented on a change in pull request #2589: [GOBBLIN-722] Unschedule gaas flow URL: https://github.com/apache/incubator-gobblin/pull/2589#discussion_r272006054 ## File path: gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/JobScheduler.java ## @@ -351,8

[jira] [Work logged] (GOBBLIN-722) add option to unschedule a gaas flow

[ https://issues.apache.org/jira/browse/GOBBLIN-722?focusedWorklogId=222782=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222782 ] ASF GitHub Bot logged work on GOBBLIN-722: -- Author: ASF GitHub Bot

[GitHub] [incubator-gobblin] jhsenjaliya commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command

jhsenjaliya commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command URL: https://github.com/apache/incubator-gobblin/pull/2578#discussion_r272026110 ## File path: conf/yarn/application.conf ## @@ -22,15 +22,18 @@

[jira] [Work logged] (GOBBLIN-707) combine & standardize all gobblin scripts into one master script & restructure configs accordingly

[ https://issues.apache.org/jira/browse/GOBBLIN-707?focusedWorklogId=222821=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222821 ] ASF GitHub Bot logged work on GOBBLIN-707: -- Author: ASF GitHub Bot

[GitHub] [incubator-gobblin] jhsenjaliya commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command

jhsenjaliya commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command URL: https://github.com/apache/incubator-gobblin/pull/2578#discussion_r272025514 ## File path: conf/yarn/reference.conf ## @@ -38,6 +38,6 @@

[jira] [Work logged] (GOBBLIN-707) combine & standardize all gobblin scripts into one master script & restructure configs accordingly

[ https://issues.apache.org/jira/browse/GOBBLIN-707?focusedWorklogId=222820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222820 ] ASF GitHub Bot logged work on GOBBLIN-707: -- Author: ASF GitHub Bot

[GitHub] [incubator-gobblin] jhsenjaliya commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command

jhsenjaliya commented on a change in pull request #2578: [GOBBLIN-707] rewrite gobblin script to combine all modes and command URL: https://github.com/apache/incubator-gobblin/pull/2578#discussion_r272026110 ## File path: conf/yarn/application.conf ## @@ -22,15 +22,18 @@