Re: [VOTE] Apache Gobblin 0.12.0 release RC3

2018-06-20 Thread Kuai Yu
+1 From: Issac Buenrostro Sent: Wednesday, June 20, 2018 8:58 AM To: dev@gobblin.incubator.apache.org Subject: Re: [VOTE] Apache Gobblin 0.12.0 release RC3 +1 On Wed, Jun 20, 2018 at 3:51 AM Abhishek Tiwari wrote: > Hi all, > > I'd like to call a vote to

Re: Read locality on gobblin jobs

2019-08-12 Thread Kuai Yu
The Helix framework in cluster mode doesn't have data locality concept. I think that is only in YARN/MR mode. From: Jay Sen Sent: Sunday, August 11, 2019 5:25 PM To: dev@gobblin.incubator.apache.org Subject: Read locality on gobblin jobs Hi Dev Team, when

Re: Read locality on gobblin jobs

2019-08-15 Thread Kuai Yu
Yes, your understanding is correct. From: Jay Sen Sent: Thursday, August 15, 2019 11:05 AM To: dev@gobblin.incubator.apache.org Subject: Re: Read locality on gobblin jobs Hi Kuai, looks like the inout to the mapreduce is the workunit file, not the actual data,

[jira] [Assigned] (GOBBLIN-19) dataset specific properties are ignored by KafkaBiLevelWorkUnitPacker

2017-08-07 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-19?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu reassigned GOBBLIN-19: -- Assignee: Kuai Yu Sprint: Apache Gobblin 170807 > dataset specific properties are igno

[jira] [Created] (GOBBLIN-214) Filtering doesn't work in FileListUtils:listFilesRecursively

2017-08-18 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-214: --- Summary: Filtering doesn't work in FileListUtils:listFilesRecursively Key: GOBBLIN-214 URL: https://issues.apache.org/jira/browse/GOBBLIN-214 Project: Apache Gobblin

[jira] [Closed] (GOBBLIN-38) Create workunitstream for CompactionSource

2017-08-17 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-38?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu closed GOBBLIN-38. -- Resolution: Fixed This is a duplicate PR. We already have a workunit stream for CompactionSource. Close

[jira] [Updated] (GOBBLIN-214) Filtering doesn't work in FileListUtils:listFilesRecursively

2017-08-18 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu updated GOBBLIN-214: Description: The filtering logic for FileListUtils:listFilesRecursively was wrong. It never applies

[jira] [Commented] (GOBBLIN-214) Filtering doesn't work in FileListUtils:listFilesRecursively

2017-08-18 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133645#comment-16133645 ] Kuai Yu commented on GOBBLIN-214: - The problem came up when we try to use this method to filter out all

[jira] [Updated] (GOBBLIN-241) Allow multiple datasets send different lineage event for kafka

2017-09-11 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu updated GOBBLIN-241: Summary: Allow multiple datasets send different lineage event for kafka (was: Add task level lineage

[jira] [Created] (GOBBLIN-252) Add some azkaban related constants

2017-09-14 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-252: --- Summary: Add some azkaban related constants Key: GOBBLIN-252 URL: https://issues.apache.org/jira/browse/GOBBLIN-252 Project: Apache Gobblin Issue Type: Improvement

[jira] [Created] (GOBBLIN-277) Add a lock to make multihop thread safe

2017-10-05 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-277: --- Summary: Add a lock to make multihop thread safe Key: GOBBLIN-277 URL: https://issues.apache.org/jira/browse/GOBBLIN-277 Project: Apache Gobblin Issue Type: Bug

[jira] [Created] (GOBBLIN-244) Need additional info for gobblin tracking hourly-deduped

2017-09-07 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-244: --- Summary: Need additional info for gobblin tracking hourly-deduped Key: GOBBLIN-244 URL: https://issues.apache.org/jira/browse/GOBBLIN-244 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-241) Add task level lineage submission for kafka lineage event support

2017-09-06 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-241: --- Summary: Add task level lineage submission for kafka lineage event support Key: GOBBLIN-241 URL: https://issues.apache.org/jira/browse/GOBBLIN-241 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-268) Unique job uri and job name generation for GaaS

2017-09-27 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-268: --- Summary: Unique job uri and job name generation for GaaS Key: GOBBLIN-268 URL: https://issues.apache.org/jira/browse/GOBBLIN-268 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-235) Prevent log warnings when TaskStateCollectorService has no task states detected

2017-09-01 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-235: --- Summary: Prevent log warnings when TaskStateCollectorService has no task states detected Key: GOBBLIN-235 URL: https://issues.apache.org/jira/browse/GOBBLIN-235 Project

[jira] [Created] (GOBBLIN-233) Add concurrent map to avoid multiple job submission from GobblinHelixJobScheduler

2017-08-31 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-233: --- Summary: Add concurrent map to avoid multiple job submission from GobblinHelixJobScheduler Key: GOBBLIN-233 URL: https://issues.apache.org/jira/browse/GOBBLIN-233 Project

[jira] [Created] (GOBBLIN-326) Gobblin metrics constructor only provides default constructor for Codhale metrics

2017-11-29 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-326: --- Summary: Gobblin metrics constructor only provides default constructor for Codhale metrics Key: GOBBLIN-326 URL: https://issues.apache.org/jira/browse/GOBBLIN-326 Project

[jira] [Created] (GOBBLIN-320) Add metrics to GobblinHelixJobScheduler

2017-11-22 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-320: --- Summary: Add metrics to GobblinHelixJobScheduler Key: GOBBLIN-320 URL: https://issues.apache.org/jira/browse/GOBBLIN-320 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-349) Add guages for gobblin cluster metrics

2017-12-15 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-349: --- Summary: Add guages for gobblin cluster metrics Key: GOBBLIN-349 URL: https://issues.apache.org/jira/browse/GOBBLIN-349 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-303) Compaction can generate zero sized output when MR is in speculative mode

2017-11-02 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-303: --- Summary: Compaction can generate zero sized output when MR is in speculative mode Key: GOBBLIN-303 URL: https://issues.apache.org/jira/browse/GOBBLIN-303 Project: Apache

[jira] [Created] (GOBBLIN-308) Gobblin cluster bootup hangs

2017-11-07 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-308: --- Summary: Gobblin cluster bootup hangs Key: GOBBLIN-308 URL: https://issues.apache.org/jira/browse/GOBBLIN-308 Project: Apache Gobblin Issue Type: Bug

[jira] [Created] (GOBBLIN-490) Allow job to be distributed by Helix

2018-05-10 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-490: --- Summary: Allow job to be distributed by Helix Key: GOBBLIN-490 URL: https://issues.apache.org/jira/browse/GOBBLIN-490 Project: Apache Gobblin Issue Type: New Feature

[jira] [Updated] (GOBBLIN-490) Allow job to be distributed by Helix

2018-05-10 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu updated GOBBLIN-490: Description: This new job launcher will submit original job to Helix instead of the generated work units

[jira] [Updated] (GOBBLIN-495) FlowSpec should be deleted if this is run once flow

2018-05-17 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu updated GOBBLIN-495: Issue Type: Bug (was: New Feature) > FlowSpec should be deleted if this is run once f

[jira] [Created] (GOBBLIN-495) FlowSpec should be deleted if this is run once flow

2018-05-17 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-495: --- Summary: FlowSpec should be deleted if this is run once flow Key: GOBBLIN-495 URL: https://issues.apache.org/jira/browse/GOBBLIN-495 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-497) GobblinHelixJobScheduler should not start scheduling before the scheduler service is up

2018-05-21 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-497: --- Summary: GobblinHelixJobScheduler should not start scheduling before the scheduler service is up Key: GOBBLIN-497 URL: https://issues.apache.org/jira/browse/GOBBLIN-497

[jira] [Created] (GOBBLIN-504) HiveMetastoreClientPool has findbugsMain issue due to unprotected static variable initialization

2018-05-24 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-504: --- Summary: HiveMetastoreClientPool has findbugsMain issue due to unprotected static variable initialization Key: GOBBLIN-504 URL: https://issues.apache.org/jira/browse/GOBBLIN-504

[jira] [Updated] (GOBBLIN-490) Add planning job execution launcher

2018-06-13 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu updated GOBBLIN-490: Summary: Add planning job execution launcher (was: Allow job to be distributed by Helix) >

[jira] [Updated] (GOBBLIN-490) Add planning job execution launcher

2018-06-13 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu updated GOBBLIN-490: Description: This new job launcher will forward the original job to one of the GobblinTaskRunner(s

[jira] [Created] (GOBBLIN-506) Job tagging support in Gobblin cluster

2018-05-29 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-506: --- Summary: Job tagging support in Gobblin cluster Key: GOBBLIN-506 URL: https://issues.apache.org/jira/browse/GOBBLIN-506 Project: Apache Gobblin Issue Type: New

[jira] [Created] (GOBBLIN-510) Decouple JobExecutionLauncher and JobExecutionDriver

2018-06-06 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-510: --- Summary: Decouple JobExecutionLauncher and JobExecutionDriver Key: GOBBLIN-510 URL: https://issues.apache.org/jira/browse/GOBBLIN-510 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-522) Multiple build issues

2018-06-29 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-522: --- Summary: Multiple build issues Key: GOBBLIN-522 URL: https://issues.apache.org/jira/browse/GOBBLIN-522 Project: Apache Gobblin Issue Type: Bug Reporter

[jira] [Created] (GOBBLIN-484) Propagate fork exception to task commit

2018-05-02 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-484: --- Summary: Propagate fork exception to task commit Key: GOBBLIN-484 URL: https://issues.apache.org/jira/browse/GOBBLIN-484 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-480) Allow job distribution cluster to be separated from cluster manager cluster

2018-04-29 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-480: --- Summary: Allow job distribution cluster to be separated from cluster manager cluster Key: GOBBLIN-480 URL: https://issues.apache.org/jira/browse/GOBBLIN-480 Project: Apache

[jira] [Updated] (GOBBLIN-480) Allow job distribution cluster to be separated from cluster manager cluster

2018-04-29 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu updated GOBBLIN-480: Description: Today GobblinClusterManager leverages single Helix cluster responsible for both job

[jira] [Created] (GOBBLIN-358) Add logs for GobblinMetrics

2018-01-05 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-358: --- Summary: Add logs for GobblinMetrics Key: GOBBLIN-358 URL: https://issues.apache.org/jira/browse/GOBBLIN-358 Project: Apache Gobblin Issue Type: Bug

[jira] [Created] (GOBBLIN-373) Expose task executor auto scale metrics to external sensor

2018-01-16 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-373: --- Summary: Expose task executor auto scale metrics to external sensor Key: GOBBLIN-373 URL: https://issues.apache.org/jira/browse/GOBBLIN-373 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-378) Task only publish data when the state is successful in the earlier processing

2018-01-17 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-378: --- Summary: Task only publish data when the state is successful in the earlier processing Key: GOBBLIN-378 URL: https://issues.apache.org/jira/browse/GOBBLIN-378 Project: Apache

[jira] [Created] (GOBBLIN-403) Fix the NPE issue due to uninitialized kafkajobmonitor metrics

2018-02-02 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-403: --- Summary: Fix the NPE issue due to uninitialized kafkajobmonitor metrics Key: GOBBLIN-403 URL: https://issues.apache.org/jira/browse/GOBBLIN-403 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-416) Allow user to configure java options to launch child process for cluster task isolation

2018-02-22 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-416: --- Summary: Allow user to configure java options to launch child process for cluster task isolation Key: GOBBLIN-416 URL: https://issues.apache.org/jira/browse/GOBBLIN-416

[jira] [Created] (GOBBLIN-356) hanging when retrieving kafka schema

2018-01-03 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-356: --- Summary: hanging when retrieving kafka schema Key: GOBBLIN-356 URL: https://issues.apache.org/jira/browse/GOBBLIN-356 Project: Apache Gobblin Issue Type: Bug

[jira] [Created] (GOBBLIN-546) Use appropriate Lists package

2018-07-30 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-546: --- Summary: Use appropriate Lists package Key: GOBBLIN-546 URL: https://issues.apache.org/jira/browse/GOBBLIN-546 Project: Apache Gobblin Issue Type: Bug

[jira] [Created] (GOBBLIN-550) When RuntimeException occurred, alwaysDelete flag doesn't work

2018-08-01 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-550: --- Summary: When RuntimeException occurred, alwaysDelete flag doesn't work Key: GOBBLIN-550 URL: https://issues.apache.org/jira/browse/GOBBLIN-550 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-536) Allow user to configure connection string properties in mysql extractor

2018-07-16 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-536: --- Summary: Allow user to configure connection string properties in mysql extractor Key: GOBBLIN-536 URL: https://issues.apache.org/jira/browse/GOBBLIN-536 Project: Apache

[jira] [Created] (GOBBLIN-574) elasticsearch-dep module failed to build

2018-09-05 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-574: --- Summary: elasticsearch-dep module failed to build Key: GOBBLIN-574 URL: https://issues.apache.org/jira/browse/GOBBLIN-574 Project: Apache Gobblin Issue Type: Bug

[jira] [Created] (GOBBLIN-575) Remove scala dependencies

2018-09-05 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-575: --- Summary: Remove scala dependencies Key: GOBBLIN-575 URL: https://issues.apache.org/jira/browse/GOBBLIN-575 Project: Apache Gobblin Issue Type: Improvement

[jira] [Created] (GOBBLIN-532) Always delete jobSpec no matter if the job is successful or not

2018-07-12 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-532: --- Summary: Always delete jobSpec no matter if the job is successful or not Key: GOBBLIN-532 URL: https://issues.apache.org/jira/browse/GOBBLIN-532 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-535) Add second hop for distributed job launcher

2018-07-13 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-535: --- Summary: Add second hop for distributed job launcher Key: GOBBLIN-535 URL: https://issues.apache.org/jira/browse/GOBBLIN-535 Project: Apache Gobblin Issue Type: New

[jira] [Commented] (GOBBLIN-373) Expose task executor auto scale metrics to external sensor

2018-01-16 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327617#comment-16327617 ] Kuai Yu commented on GOBBLIN-373: - [~jbaranick], we are using *StandardMetricsBridge* interface to expose

[jira] [Created] (GOBBLIN-584) Fix the helix key configuration naming.

2018-09-11 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-584: --- Summary: Fix the helix key configuration naming. Key: GOBBLIN-584 URL: https://issues.apache.org/jira/browse/GOBBLIN-584 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-419) Add more metrics for cluster job scheduling

2018-02-27 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-419: --- Summary: Add more metrics for cluster job scheduling Key: GOBBLIN-419 URL: https://issues.apache.org/jira/browse/GOBBLIN-419 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-423) Limit records or bucket counts for dynamic probing

2018-03-08 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-423: --- Summary: Limit records or bucket counts for dynamic probing Key: GOBBLIN-423 URL: https://issues.apache.org/jira/browse/GOBBLIN-423 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-466) Reuse same connector for Salesforce dynamic partitioning

2018-04-13 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-466: --- Summary: Reuse same connector for Salesforce dynamic partitioning Key: GOBBLIN-466 URL: https://issues.apache.org/jira/browse/GOBBLIN-466 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-473) Allow user to configure different lookback time for different datasets

2018-04-24 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-473: --- Summary: Allow user to configure different lookback time for different datasets Key: GOBBLIN-473 URL: https://issues.apache.org/jira/browse/GOBBLIN-473 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-476) Add helix task timeout

2018-04-25 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-476: --- Summary: Add helix task timeout Key: GOBBLIN-476 URL: https://issues.apache.org/jira/browse/GOBBLIN-476 Project: Apache Gobblin Issue Type: Improvement

[jira] [Created] (GOBBLIN-448) Add glob pattern blacklist in ConfigurableGlobDatasetFinder

2018-03-27 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-448: --- Summary: Add glob pattern blacklist in ConfigurableGlobDatasetFinder Key: GOBBLIN-448 URL: https://issues.apache.org/jira/browse/GOBBLIN-448 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-445) Add task output directory for staging compaction result

2018-03-26 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-445: --- Summary: Add task output directory for staging compaction result Key: GOBBLIN-445 URL: https://issues.apache.org/jira/browse/GOBBLIN-445 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-436) Salesforce doesn't have default constructor

2018-03-22 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-436: --- Summary: Salesforce doesn't have default constructor Key: GOBBLIN-436 URL: https://issues.apache.org/jira/browse/GOBBLIN-436 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-602) Allow AzkabanProducer to be customized

2018-10-05 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-602: --- Summary: Allow AzkabanProducer to be customized Key: GOBBLIN-602 URL: https://issues.apache.org/jira/browse/GOBBLIN-602 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-615) Make LWM==HWM a valid interval in QueryBaseSource

2018-10-19 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-615: --- Summary: Make LWM==HWM a valid interval in QueryBaseSource Key: GOBBLIN-615 URL: https://issues.apache.org/jira/browse/GOBBLIN-615 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-617) Add distributed job launcher metrics and some refactoring.

2018-10-23 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-617: --- Summary: Add distributed job launcher metrics and some refactoring. Key: GOBBLIN-617 URL: https://issues.apache.org/jira/browse/GOBBLIN-617 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-619) Fix the metrics name for GobblinHelixJobScheduler

2018-10-24 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-619: --- Summary: Fix the metrics name for GobblinHelixJobScheduler Key: GOBBLIN-619 URL: https://issues.apache.org/jira/browse/GOBBLIN-619 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-622) Avoid to serialize all previous workunits in SourceState to save both memory and diskspace

2018-10-26 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-622: --- Summary: Avoid to serialize all previous workunits in SourceState to save both memory and diskspace Key: GOBBLIN-622 URL: https://issues.apache.org/jira/browse/GOBBLIN-622

[jira] [Created] (GOBBLIN-620) NPE when catalog metrics is not enabled

2018-10-25 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-620: --- Summary: NPE when catalog metrics is not enabled Key: GOBBLIN-620 URL: https://issues.apache.org/jira/browse/GOBBLIN-620 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-618) Remove unnecessary methods from StandardMetricsBridge

2018-10-23 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-618: --- Summary: Remove unnecessary methods from StandardMetricsBridge Key: GOBBLIN-618 URL: https://issues.apache.org/jira/browse/GOBBLIN-618 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-625) Distributed job launcher doesn't have helix tagging support

2018-11-05 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-625: --- Summary: Distributed job launcher doesn't have helix tagging support Key: GOBBLIN-625 URL: https://issues.apache.org/jira/browse/GOBBLIN-625 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-626) Fix the wrong planning job tag key

2018-11-05 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-626: --- Summary: Fix the wrong planning job tag key Key: GOBBLIN-626 URL: https://issues.apache.org/jira/browse/GOBBLIN-626 Project: Apache Gobblin Issue Type: Bug

[jira] [Created] (GOBBLIN-601) Add cancellation to AzkabanClient

2018-10-03 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-601: --- Summary: Add cancellation to AzkabanClient Key: GOBBLIN-601 URL: https://issues.apache.org/jira/browse/GOBBLIN-601 Project: Apache Gobblin Issue Type: Improvement

[jira] [Created] (GOBBLIN-608) Allow user to configure the fork operation tiimeout

2018-10-09 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-608: --- Summary: Allow user to configure the fork operation tiimeout Key: GOBBLIN-608 URL: https://issues.apache.org/jira/browse/GOBBLIN-608 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-655) Allow helix jobs to have job type set

2018-12-13 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-655: --- Summary: Allow helix jobs to have job type set Key: GOBBLIN-655 URL: https://issues.apache.org/jira/browse/GOBBLIN-655 Project: Apache Gobblin Issue Type: Improvement

[jira] [Updated] (GOBBLIN-655) Allow helix jobs to have job type set

2018-12-13 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu updated GOBBLIN-655: Description: This is required because Helix will use job type for the quota assignment. For job

[jira] [Created] (GOBBLIN-649) Add task driver cluster

2018-12-10 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-649: --- Summary: Add task driver cluster Key: GOBBLIN-649 URL: https://issues.apache.org/jira/browse/GOBBLIN-649 Project: Apache Gobblin Issue Type: Improvement

[jira] [Created] (GOBBLIN-652) gobblin cluster doesn't have helix related metrics

2018-12-11 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-652: --- Summary: gobblin cluster doesn't have helix related metrics Key: GOBBLIN-652 URL: https://issues.apache.org/jira/browse/GOBBLIN-652 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-647) Move early stop feature to task driver

2018-12-06 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-647: --- Summary: Move early stop feature to task driver Key: GOBBLIN-647 URL: https://issues.apache.org/jira/browse/GOBBLIN-647 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-639) Change serder method to static from RequesterService

2018-11-27 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-639: --- Summary: Change serder method to static from RequesterService Key: GOBBLIN-639 URL: https://issues.apache.org/jira/browse/GOBBLIN-639 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-661) Prevent jobs resubmission after manager failure

2019-01-08 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-661: --- Summary: Prevent jobs resubmission after manager failure Key: GOBBLIN-661 URL: https://issues.apache.org/jira/browse/GOBBLIN-661 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-664) AzkabanClient session refresh logic is not configurable

2019-01-11 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-664: --- Summary: AzkabanClient session refresh logic is not configurable Key: GOBBLIN-664 URL: https://issues.apache.org/jira/browse/GOBBLIN-664 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-591) Allow user to pass in the customized http client for azkaban client

2018-09-17 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-591: --- Summary: Allow user to pass in the customized http client for azkaban client Key: GOBBLIN-591 URL: https://issues.apache.org/jira/browse/GOBBLIN-591 Project: Apache Gobblin

[jira] [Created] (GOBBLIN-588) Remove final decorator to allow password be overwritten

2018-09-13 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-588: --- Summary: Remove final decorator to allow password be overwritten Key: GOBBLIN-588 URL: https://issues.apache.org/jira/browse/GOBBLIN-588 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-712) Add version strategy for configbased dataset copy

2019-03-25 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-712: --- Summary: Add version strategy for configbased dataset copy Key: GOBBLIN-712 URL: https://issues.apache.org/jira/browse/GOBBLIN-712 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-713) Lazy load job specification from job catalog to avoid OOM issue when JobCatalog is bootup.

2019-03-26 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-713: --- Summary: Lazy load job specification from job catalog to avoid OOM issue when JobCatalog is bootup. Key: GOBBLIN-713 URL: https://issues.apache.org/jira/browse/GOBBLIN-713

[jira] [Created] (GOBBLIN-690) Relaunch check for the planning job is not correct

2019-02-25 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-690: --- Summary: Relaunch check for the planning job is not correct Key: GOBBLIN-690 URL: https://issues.apache.org/jira/browse/GOBBLIN-690 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-703) Allow planning job to be run in a non-blocking way

2019-03-19 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-703: --- Summary: Allow planning job to be run in a non-blocking way Key: GOBBLIN-703 URL: https://issues.apache.org/jira/browse/GOBBLIN-703 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-683) Azkaban client should retry if session gets expired

2019-02-14 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-683: --- Summary: Azkaban client should retry if session gets expired Key: GOBBLIN-683 URL: https://issues.apache.org/jira/browse/GOBBLIN-683 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-729) Add version strategy support for HiveDataset copy

2019-04-08 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-729: --- Summary: Add version strategy support for HiveDataset copy Key: GOBBLIN-729 URL: https://issues.apache.org/jira/browse/GOBBLIN-729 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-783) Fix the double referencing issue for job type config

2019-05-24 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-783: --- Summary: Fix the double referencing issue for job type config Key: GOBBLIN-783 URL: https://issues.apache.org/jira/browse/GOBBLIN-783 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-813) Make SFDC connector support encrypted Salesforce client id and client secret

2019-06-24 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-813: --- Summary: Make SFDC connector support encrypted Salesforce client id and client secret Key: GOBBLIN-813 URL: https://issues.apache.org/jira/browse/GOBBLIN-813 Project: Apache

[jira] [Created] (GOBBLIN-751) Make enforced file size matching to be configurable

2019-04-23 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-751: --- Summary: Make enforced file size matching to be configurable Key: GOBBLIN-751 URL: https://issues.apache.org/jira/browse/GOBBLIN-751 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-851) Provide capability to disable hive schema registration in partition level

2019-08-12 Thread Kuai Yu (JIRA)
Kuai Yu created GOBBLIN-851: --- Summary: Provide capability to disable hive schema registration in partition level Key: GOBBLIN-851 URL: https://issues.apache.org/jira/browse/GOBBLIN-851 Project: Apache

[jira] [Updated] (GOBBLIN-851) Provide capability to disable hive schema registration in partition level

2019-08-12 Thread Kuai Yu (JIRA)
[ https://issues.apache.org/jira/browse/GOBBLIN-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu updated GOBBLIN-851: Description: We had problems when table level schema and partition level schema diverges. Think about

[jira] [Created] (GOBBLIN-889) FileContext doesn't use the correct writer fs uri

2019-09-25 Thread Kuai Yu (Jira)
Kuai Yu created GOBBLIN-889: --- Summary: FileContext doesn't use the correct writer fs uri Key: GOBBLIN-889 URL: https://issues.apache.org/jira/browse/GOBBLIN-889 Project: Apache Gobblin Issue Type

[jira] [Created] (GOBBLIN-962) Refactor RecursiveCopyableDataset so that the copy entities generation logic can be reused.

2019-11-14 Thread Kuai Yu (Jira)
Kuai Yu created GOBBLIN-962: --- Summary: Refactor RecursiveCopyableDataset so that the copy entities generation logic can be reused. Key: GOBBLIN-962 URL: https://issues.apache.org/jira/browse/GOBBLIN-962

[jira] [Created] (GOBBLIN-915) Extract cannot parse the timezone

2019-10-17 Thread Kuai Yu (Jira)
Kuai Yu created GOBBLIN-915: --- Summary: Extract cannot parse the timezone Key: GOBBLIN-915 URL: https://issues.apache.org/jira/browse/GOBBLIN-915 Project: Apache Gobblin Issue Type: Bug

[jira] [Created] (GOBBLIN-1049) Move workunit commit logic to the end of publish().

2020-02-13 Thread Kuai Yu (Jira)
Kuai Yu created GOBBLIN-1049: Summary: Move workunit commit logic to the end of publish(). Key: GOBBLIN-1049 URL: https://issues.apache.org/jira/browse/GOBBLIN-1049 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-1129) clean up staging table created by avro2orc pipeline

2020-04-27 Thread Kuai Yu (Jira)
Kuai Yu created GOBBLIN-1129: Summary: clean up staging table created by avro2orc pipeline Key: GOBBLIN-1129 URL: https://issues.apache.org/jira/browse/GOBBLIN-1129 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-1124) Error message should include throwable details when http converter hits a failure

2020-04-21 Thread Kuai Yu (Jira)
Kuai Yu created GOBBLIN-1124: Summary: Error message should include throwable details when http converter hits a failure Key: GOBBLIN-1124 URL: https://issues.apache.org/jira/browse/GOBBLIN-1124 Project

[jira] [Created] (GOBBLIN-1134) Improve FAQs

2020-04-29 Thread Kuai Yu (Jira)
Kuai Yu created GOBBLIN-1134: Summary: Improve FAQs Key: GOBBLIN-1134 URL: https://issues.apache.org/jira/browse/GOBBLIN-1134 Project: Apache Gobblin Issue Type: Improvement Reporter

[jira] [Resolved] (GOBBLIN-1124) Error message should include throwable details when http converter hits a failure

2020-04-22 Thread Kuai Yu (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu resolved GOBBLIN-1124. -- Resolution: Fixed > Error message should include throwable details when http converter h

[jira] [Resolved] (GOBBLIN-277) Add a lock to make multihop thread safe

2020-07-24 Thread Kuai Yu (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu resolved GOBBLIN-277. - Resolution: Fixed > Add a lock to make multihop thread s

[jira] [Resolved] (GOBBLIN-373) Expose task executor auto scale metrics to external sensor

2020-07-24 Thread Kuai Yu (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuai Yu resolved GOBBLIN-373. - Resolution: Fixed > Expose task executor auto scale metrics to external sen

  1   2   >