[jira] [Created] (GOBBLIN-1213) Skip job state deserialization during SingleFailInCreationTask instantiation

2020-07-15 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1213: Summary: Skip job state deserialization during SingleFailInCreationTask instantiation Key: GOBBLIN-1213 URL: https://issues.apache.org/jira/browse/GOBBLIN-1213

[jira] [Resolved] (GOBBLIN-1211) Track unused Helix instances in a thread-safe manner

2020-07-14 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1211. -- Resolution: Fixed > Track unused Helix instances in a thread-safe manner >

[jira] [Closed] (GOBBLIN-1211) Track unused Helix instances in a thread-safe manner

2020-07-14 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1211. > Track unused Helix instances in a thread-safe manner >

[jira] [Created] (GOBBLIN-1211) Track unused Helix instances in a thread-safe manner

2020-07-13 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1211: Summary: Track unused Helix instances in a thread-safe manner Key: GOBBLIN-1211 URL: https://issues.apache.org/jira/browse/GOBBLIN-1211 Project:

[jira] [Created] (GOBBLIN-1209) Provide an option to configure the java tmp dir to the Yarn cache location

2020-07-07 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1209: Summary: Provide an option to configure the java tmp dir to the Yarn cache location Key: GOBBLIN-1209 URL: https://issues.apache.org/jira/browse/GOBBLIN-1209

[jira] [Created] (GOBBLIN-1193) Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor

2020-06-12 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1193: Summary: Ensure that ingestion latency is 0 when no records are consumed by Kafka Extractor Key: GOBBLIN-1193 URL:

[jira] [Created] (GOBBLIN-1191) Reuse Helix instance names when containers are released by Gobblin Application Master

2020-06-11 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1191: Summary: Reuse Helix instance names when containers are released by Gobblin Application Master Key: GOBBLIN-1191 URL:

[jira] [Created] (GOBBLIN-1189) Relax the condition for the increasing ingestion latency check

2020-06-10 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1189: Summary: Relax the condition for the increasing ingestion latency check Key: GOBBLIN-1189 URL: https://issues.apache.org/jira/browse/GOBBLIN-1189

[jira] [Created] (GOBBLIN-1177) Provide a config for overprovisioning Gobblin Yarn containers by a configurable amount

2020-06-02 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1177: Summary: Provide a config for overprovisioning Gobblin Yarn containers by a configurable amount Key: GOBBLIN-1177 URL:

[jira] [Created] (GOBBLIN-1175) Provide an option to all GobblinYarnAppLauncher to detach from Yarn application

2020-06-01 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1175: Summary: Provide an option to all GobblinYarnAppLauncher to detach from Yarn application Key: GOBBLIN-1175 URL: https://issues.apache.org/jira/browse/GOBBLIN-1175

[jira] [Created] (GOBBLIN-1173) Prevent propagating exceptions from the finally block in KafkaSource#getWorkUnits

2020-06-01 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1173: Summary: Prevent propagating exceptions from the finally block in KafkaSource#getWorkUnits Key: GOBBLIN-1173 URL:

[jira] [Created] (GOBBLIN-1162) Provide an option to allow slow containers to commit suicide when unhealthy

2020-05-27 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1162: Summary: Provide an option to allow slow containers to commit suicide when unhealthy Key: GOBBLIN-1162 URL: https://issues.apache.org/jira/browse/GOBBLIN-1162

[jira] [Created] (GOBBLIN-1153) Revert changes in PR 2967

2020-05-20 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1153: Summary: Revert changes in PR 2967 Key: GOBBLIN-1153 URL: https://issues.apache.org/jira/browse/GOBBLIN-1153 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-1127) Provide an option to make metric reporting failure fatal

2020-04-24 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1127: Summary: Provide an option to make metric reporting failure fatal Key: GOBBLIN-1127 URL: https://issues.apache.org/jira/browse/GOBBLIN-1127 Project:

[jira] [Updated] (GOBBLIN-1127) Provide an option to make metric reporting instantiation failures fatal

2020-04-24 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan updated GOBBLIN-1127: - Summary: Provide an option to make metric reporting instantiation failures

[jira] [Created] (GOBBLIN-1125) Add metrics to measure job status state store performance in Gobblin Service

2020-04-22 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1125: Summary: Add metrics to measure job status state store performance in Gobblin Service Key: GOBBLIN-1125 URL: https://issues.apache.org/jira/browse/GOBBLIN-1125

[jira] [Created] (GOBBLIN-1123) Report orchestration delay for Gobblin Service flows

2020-04-21 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1123: Summary: Report orchestration delay for Gobblin Service flows Key: GOBBLIN-1123 URL: https://issues.apache.org/jira/browse/GOBBLIN-1123 Project:

[jira] [Created] (GOBBLIN-1122) Bump up helix-lib version

2020-04-20 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1122: Summary: Bump up helix-lib version Key: GOBBLIN-1122 URL: https://issues.apache.org/jira/browse/GOBBLIN-1122 Project: Apache Gobblin Issue

[jira] [Created] (GOBBLIN-1120) Reinitialize HelixManager when Helix participant check throws an exception when reading

2020-04-16 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1120: Summary: Reinitialize HelixManager when Helix participant check throws an exception when reading Key: GOBBLIN-1120 URL:

[jira] [Updated] (GOBBLIN-1120) Reinitialize HelixManager when Helix participant check throws an exception

2020-04-16 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan updated GOBBLIN-1120: - Summary: Reinitialize HelixManager when Helix participant check throws an

[jira] [Created] (GOBBLIN-1116) Avoid registering schema with schema registry during MetricReporting initialization from cluster workers

2020-04-13 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1116: Summary: Avoid registering schema with schema registry during MetricReporting initialization from cluster workers Key: GOBBLIN-1116 URL:

[jira] [Updated] (GOBBLIN-1112) Implement a new HttpMethodRetryHandler that allows retrying a HTTP method on transient network errors

2020-04-09 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan updated GOBBLIN-1112: - Summary: Implement a new HttpMethodRetryHandler that allows retrying a HTTP

[jira] [Created] (GOBBLIN-1112) Create a new HttpMethodRetryHandler implementation that allows retrying a HTTP method on transient network errors

2020-04-09 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1112: Summary: Create a new HttpMethodRetryHandler implementation that allows retrying a HTTP method on transient network errors Key: GOBBLIN-1112 URL:

[jira] [Closed] (GOBBLIN-1052) Create a spec consumer path if it does not exist in FS SpecConsumer

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1052. > Create a spec consumer path if it does not exist in FS SpecConsumer >

[jira] [Closed] (GOBBLIN-1076) Make Gobblin cluster working directories configurable

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1076. > Make Gobblin cluster working directories configurable >

[jira] [Resolved] (GOBBLIN-1067) Add SFTP DataNode type in Gobblin-as-a-Service (GaaS) FlowGraph

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1067. -- Resolution: Fixed > Add SFTP DataNode type in Gobblin-as-a-Service (GaaS)

[jira] [Resolved] (GOBBLIN-1052) Create a spec consumer path if it does not exist in FS SpecConsumer

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1052. -- Resolution: Fixed > Create a spec consumer path if it does not exist in FS

[jira] [Closed] (GOBBLIN-1067) Add SFTP DataNode type in Gobblin-as-a-Service (GaaS) FlowGraph

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1067. > Add SFTP DataNode type in Gobblin-as-a-Service (GaaS) FlowGraph >

[jira] [Closed] (GOBBLIN-1087) Track and report histogram of observed lag from Gobblin Kafka pipeline

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1087. > Track and report histogram of observed lag from Gobblin Kafka pipeline >

[jira] [Resolved] (GOBBLIN-1099) Handle orphaned Yarn containers in Gobblin-on-Yarn clusters

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1099. -- Resolution: Fixed > Handle orphaned Yarn containers in Gobblin-on-Yarn

[jira] [Resolved] (GOBBLIN-1091) Pass Yarn application id as part of AppMaster and YarnTaskRunner's start up command

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1091. -- Resolution: Fixed > Pass Yarn application id as part of AppMaster and

[jira] [Closed] (GOBBLIN-1099) Handle orphaned Yarn containers in Gobblin-on-Yarn clusters

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1099. > Handle orphaned Yarn containers in Gobblin-on-Yarn clusters >

[jira] [Closed] (GOBBLIN-1091) Pass Yarn application id as part of AppMaster and YarnTaskRunner's start up command

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1091. > Pass Yarn application id as part of AppMaster and YarnTaskRunner's start up >

[jira] [Resolved] (GOBBLIN-1076) Make Gobblin cluster working directories configurable

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1076. -- Resolution: Fixed > Make Gobblin cluster working directories configurable >

[jira] [Resolved] (GOBBLIN-1087) Track and report histogram of observed lag from Gobblin Kafka pipeline

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1087. -- Resolution: Fixed > Track and report histogram of observed lag from Gobblin

[jira] [Closed] (GOBBLIN-1107) Lazily initialize Helix TaskStateModelFactory in GobblinTaskRunner

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1107. > Lazily initialize Helix TaskStateModelFactory in GobblinTaskRunner >

[jira] [Resolved] (GOBBLIN-1107) Lazily initialize Helix TaskStateModelFactory in GobblinTaskRunner

2020-04-03 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1107. -- Resolution: Fixed > Lazily initialize Helix TaskStateModelFactory in

[jira] [Created] (GOBBLIN-1107) Lazily initialize Helix TaskStateModelFactory in GobblinTaskRunner

2020-04-01 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1107: Summary: Lazily initialize Helix TaskStateModelFactory in GobblinTaskRunner Key: GOBBLIN-1107 URL: https://issues.apache.org/jira/browse/GOBBLIN-1107

[jira] [Created] (GOBBLIN-1099) Handle orphaned Yarn containers in Gobblin-on-Yarn clusters

2020-03-26 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1099: Summary: Handle orphaned Yarn containers in Gobblin-on-Yarn clusters Key: GOBBLIN-1099 URL: https://issues.apache.org/jira/browse/GOBBLIN-1099

[jira] [Created] (GOBBLIN-1091) Pass Yarn application id as part of AppMaster and YarnTaskRunner's start up command

2020-03-19 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1091: Summary: Pass Yarn application id as part of AppMaster and YarnTaskRunner's start up command Key: GOBBLIN-1091 URL:

[jira] [Created] (GOBBLIN-1087) Track and report histogram of observed lag from Gobblin Kafka pipeline

2020-03-16 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1087: Summary: Track and report histogram of observed lag from Gobblin Kafka pipeline Key: GOBBLIN-1087 URL: https://issues.apache.org/jira/browse/GOBBLIN-1087

[jira] [Created] (GOBBLIN-1076) Make Gobblin cluster working directories configurable

2020-03-10 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1076: Summary: Make Gobblin cluster working directories configurable Key: GOBBLIN-1076 URL: https://issues.apache.org/jira/browse/GOBBLIN-1076 Project:

[jira] [Created] (GOBBLIN-1067) Add SFTP DataNode type in Gobblin-as-a-Service (GaaS) FlowGraph

2020-03-03 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1067: Summary: Add SFTP DataNode type in Gobblin-as-a-Service (GaaS) FlowGraph Key: GOBBLIN-1067 URL: https://issues.apache.org/jira/browse/GOBBLIN-1067

[jira] [Resolved] (GOBBLIN-1048) Provide an option to pass and set System properties via Gobblin Cluster application config

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1048. -- Resolution: Fixed > Provide an option to pass and set System properties via

[jira] [Closed] (GOBBLIN-1048) Provide an option to pass and set System properties via Gobblin Cluster application config

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1048. > Provide an option to pass and set System properties via Gobblin Cluster >

[jira] [Closed] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1016. > Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode > when

[jira] [Closed] (GOBBLIN-1018) Report GC counts and durations from Gobblin containers metrics service

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1018. > Report GC counts and durations from Gobblin containers metrics service >

[jira] [Resolved] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1016. -- Resolution: Fixed > Allow Gobblin Application Master to join Helix cluster

[jira] [Closed] (GOBBLIN-1029) Maintain the last GC stats to accurately report the difference in each interval

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1029. > Maintain the last GC stats to accurately report the difference in each > interval

[jira] [Closed] (GOBBLIN-1030) Refactor AbstractYarnSecurityManager to expose method for sending token file update message

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1030. > Refactor AbstractYarnSecurityManager to expose method for sending token file >

[jira] [Closed] (GOBBLIN-1036) Add hadoop override configurations when instantiating FileSystem object in GobblinTaskRunner and GobblinClusterManager

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1036. > Add hadoop override configurations when instantiating FileSystem object in >

[jira] [Resolved] (GOBBLIN-1030) Refactor AbstractYarnSecurityManager to expose method for sending token file update message

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1030. -- Resolution: Fixed > Refactor AbstractYarnSecurityManager to expose method

[jira] [Resolved] (GOBBLIN-1036) Add hadoop override configurations when instantiating FileSystem object in GobblinTaskRunner and GobblinClusterManager

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1036. -- Resolution: Fixed > Add hadoop override configurations when instantiating

[jira] [Resolved] (GOBBLIN-1032) Provide Helix instance tags config to GobblinYarnTaskRunner during container start up

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1032. -- Resolution: Fixed > Provide Helix instance tags config to

[jira] [Closed] (GOBBLIN-1032) Provide Helix instance tags config to GobblinYarnTaskRunner during container start up

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1032. > Provide Helix instance tags config to GobblinYarnTaskRunner during container >

[jira] [Resolved] (GOBBLIN-1018) Report GC counts and durations from Gobblin containers metrics service

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1018. -- Resolution: Fixed > Report GC counts and durations from Gobblin containers

[jira] [Closed] (GOBBLIN-1034) Ensure underlying writers are expired from the PartitionedDataWriter cache to avoid accumulation of writers for long running Gobblin jobs

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1034. > Ensure underlying writers are expired from the PartitionedDataWriter cache to >

[jira] [Resolved] (GOBBLIN-1029) Maintain the last GC stats to accurately report the difference in each interval

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1029. -- Resolution: Fixed > Maintain the last GC stats to accurately report the

[jira] [Resolved] (GOBBLIN-1043) Implement a Helix assigned participant check as a CommitStep

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1043. -- Resolution: Fixed > Implement a Helix assigned participant check as a

[jira] [Resolved] (GOBBLIN-1047) Add Helix and Yarn container metadata to all task events emitted by Gobblin Helix tasks

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1047. -- Resolution: Fixed > Add Helix and Yarn container metadata to all task events

[jira] [Closed] (GOBBLIN-1043) Implement a Helix assigned participant check as a CommitStep

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1043. > Implement a Helix assigned participant check as a CommitStep >

[jira] [Closed] (GOBBLIN-1047) Add Helix and Yarn container metadata to all task events emitted by Gobblin Helix tasks

2020-02-12 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1047. > Add Helix and Yarn container metadata to all task events emitted by Gobblin >

[jira] [Created] (GOBBLIN-1048) Provide an option to pass and set System properties via Gobblin Cluster application config

2020-02-12 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1048: Summary: Provide an option to pass and set System properties via Gobblin Cluster application config Key: GOBBLIN-1048 URL:

[jira] [Created] (GOBBLIN-1047) Add Helix and Yarn container metadata to all task events emitted by Gobblin Helix tasks

2020-02-11 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1047: Summary: Add Helix and Yarn container metadata to all task events emitted by Gobblin Helix tasks Key: GOBBLIN-1047 URL:

[jira] [Created] (GOBBLIN-1043) Implement a Helix assigned participant check as a CommitStep

2020-02-06 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1043: Summary: Implement a Helix assigned participant check as a CommitStep Key: GOBBLIN-1043 URL: https://issues.apache.org/jira/browse/GOBBLIN-1043

[jira] [Created] (GOBBLIN-1036) Add hadoop override configurations when instantiating FileSystem object in GobblinTaskRunner and GobblinClusterManager

2020-01-30 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1036: Summary: Add hadoop override configurations when instantiating FileSystem object in GobblinTaskRunner and GobblinClusterManager Key: GOBBLIN-1036 URL:

[jira] [Created] (GOBBLIN-1034) Ensure underlying writers are expired from the PartitionedDataWriter cache to avoid accumulation of writers for long running Gobblin jobs

2020-01-28 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1034: Summary: Ensure underlying writers are expired from the PartitionedDataWriter cache to avoid accumulation of writers for long running Gobblin jobs Key: GOBBLIN-1034

[jira] [Created] (GOBBLIN-1032) Provide Helix instance tags config to GobblinYarnTaskRunner during container start up

2020-01-24 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1032: Summary: Provide Helix instance tags config to GobblinYarnTaskRunner during container start up Key: GOBBLIN-1032 URL:

[jira] [Updated] (GOBBLIN-1030) Refactor AbstractYarnSecurityManager to expose method for sending token file update message

2020-01-16 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan updated GOBBLIN-1030: - Description: Current implementation has this method in

[jira] [Updated] (GOBBLIN-1030) Refactor AbstractYarnSecurityManager to expose method for sending token file update message

2020-01-16 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan updated GOBBLIN-1030: - Summary: Refactor AbstractYarnSecurityManager to expose method for sending

[jira] [Created] (GOBBLIN-1030) Expose helixClusterManaged config in AbstractYarnSecurityManager

2020-01-16 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1030: Summary: Expose helixClusterManaged config in AbstractYarnSecurityManager Key: GOBBLIN-1030 URL: https://issues.apache.org/jira/browse/GOBBLIN-1030

[jira] [Created] (GOBBLIN-1018) Report GC counts and durations from Gobblin containers metrics service

2020-01-12 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1018: Summary: Report GC counts and durations from Gobblin containers metrics service Key: GOBBLIN-1018 URL: https://issues.apache.org/jira/browse/GOBBLIN-1018

[jira] [Updated] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed

2020-01-09 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan updated GOBBLIN-1016: - Summary: Allow Gobblin Application Master to join Helix cluster in PARTICIPANT

[jira] [Updated] (GOBBLIN-1016) Allow Gobblin Application Master to join Helix cluster in PARTICIPANT mode when Helix cluster is managed.

2020-01-09 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan updated GOBBLIN-1016: - Summary: Allow Gobblin Application Master to join Helix cluster in PARTICIPANT

[jira] [Updated] (GOBBLIN-1016) Send shutdown request and token file updated messages to PARTICIPANT instance type when Helix cluster is managed

2020-01-09 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan updated GOBBLIN-1016: - Description: In the managed cluster mode, the Gobblin Yarn Application Master

[jira] [Updated] (GOBBLIN-1016) Send shutdown request and token file updated messages to PARTICIPANT instance type when Helix cluster is managed

2020-01-09 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan updated GOBBLIN-1016: - Summary: Send shutdown request and token file updated messages to PARTICIPANT

[jira] [Created] (GOBBLIN-1016) Send shutdown request and token file updated messages to ADMINISTRATOR instance type when Helix cluster is managed

2020-01-08 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1016: Summary: Send shutdown request and token file updated messages to ADMINISTRATOR instance type when Helix cluster is managed Key: GOBBLIN-1016 URL:

[jira] [Closed] (GOBBLIN-1002) Set state id when deserializing state from Gobblin state store

2019-12-10 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1002. > Set state id when deserializing state from Gobblin state store >

[jira] [Resolved] (GOBBLIN-1002) Set state id when deserializing state from Gobblin state store

2019-12-10 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1002. -- Resolution: Fixed > Set state id when deserializing state from Gobblin state

[jira] [Created] (GOBBLIN-1002) Set state id when deserializing state from Gobblin state store

2019-12-10 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1002: Summary: Set state id when deserializing state from Gobblin state store Key: GOBBLIN-1002 URL: https://issues.apache.org/jira/browse/GOBBLIN-1002

[jira] [Resolved] (GOBBLIN-1000) Add min and max LogAppendTime to tracking events emitted from Gobblin Kafka Extractor

2019-12-10 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-1000. -- Resolution: Fixed > Add min and max LogAppendTime to tracking events emitted

[jira] [Closed] (GOBBLIN-1000) Add min and max LogAppendTime to tracking events emitted from Gobblin Kafka Extractor

2019-12-10 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-1000. > Add min and max LogAppendTime to tracking events emitted from Gobblin Kafka >

[jira] [Created] (GOBBLIN-1000) Add min and max LogAppendTime to tracking events emitted from Gobblin Kafka Extractor

2019-12-09 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-1000: Summary: Add min and max LogAppendTime to tracking events emitted from Gobblin Kafka Extractor Key: GOBBLIN-1000 URL:

[jira] [Closed] (GOBBLIN-996) Add support for managed Helix clusters for Gobblin-on-Yarn applications

2019-12-05 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-996. --- > Add support for managed Helix clusters for Gobblin-on-Yarn applications >

[jira] [Resolved] (GOBBLIN-996) Add support for managed Helix clusters for Gobblin-on-Yarn applications

2019-12-05 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-996. - Resolution: Fixed > Add support for managed Helix clusters for Gobblin-on-Yarn

[jira] [Resolved] (GOBBLIN-989) Track and report record level SLA in Gobblin Kafka Extractor tracking event

2019-12-05 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-989. - Resolution: Fixed > Track and report record level SLA in Gobblin Kafka

[jira] [Closed] (GOBBLIN-989) Track and report record level SLA in Gobblin Kafka Extractor tracking event

2019-12-05 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-989. --- > Track and report record level SLA in Gobblin Kafka Extractor tracking event >

[jira] [Created] (GOBBLIN-996) Add support for managed Helix clusters for Gobblin-on-Yarn applications

2019-12-05 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-996: --- Summary: Add support for managed Helix clusters for Gobblin-on-Yarn applications Key: GOBBLIN-996 URL: https://issues.apache.org/jira/browse/GOBBLIN-996

[jira] [Created] (GOBBLIN-989) Track and report record level SLA in Gobblin Kafka Extractor tracking event

2019-12-02 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-989: --- Summary: Track and report record level SLA in Gobblin Kafka Extractor tracking event Key: GOBBLIN-989 URL: https://issues.apache.org/jira/browse/GOBBLIN-989

[jira] [Closed] (GOBBLIN-978) Use job start time instead of flow start time to kill jobs stuck in ORCHESTRATED state

2019-11-25 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-978. --- > Use job start time instead of flow start time to kill jobs stuck in > ORCHESTRATED

[jira] [Closed] (GOBBLIN-976) Add dynamic config to the state before instantiating metrics reporter in MRJobLauncher

2019-11-25 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-976. --- > Add dynamic config to the state before instantiating metrics reporter in >

[jira] [Resolved] (GOBBLIN-978) Use job start time instead of flow start time to kill jobs stuck in ORCHESTRATED state

2019-11-25 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-978. - Resolution: Fixed > Use job start time instead of flow start time to kill jobs

[jira] [Resolved] (GOBBLIN-976) Add dynamic config to the state before instantiating metrics reporter in MRJobLauncher

2019-11-25 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-976. - Resolution: Fixed > Add dynamic config to the state before instantiating

[jira] [Created] (GOBBLIN-978) Use job start time instead of flow start time to kill jobs stuck in ORCHESTRATED state

2019-11-24 Thread Sudarshan Vasudevan (Jira)
Sudarshan Vasudevan created GOBBLIN-978: --- Summary: Use job start time instead of flow start time to kill jobs stuck in ORCHESTRATED state Key: GOBBLIN-978 URL:

[jira] [Closed] (GOBBLIN-878) Implement a Metric Regex filter based on whitelist/blacklist patterns.

2019-11-21 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-878. --- Resolution: Won't Fix > Implement a Metric Regex filter based on

[jira] [Resolved] (GOBBLIN-955) Expose a method to get average record size in KafkaExtractorStatsTracker

2019-11-21 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan resolved GOBBLIN-955. - Resolution: Fixed > Expose a method to get average record size in

[jira] [Closed] (GOBBLIN-875) Emit container health metrics when running in cluster mode

2019-11-21 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-875. --- > Emit container health metrics when running in cluster mode >

[jira] [Closed] (GOBBLIN-945) Refactor Kafka extractor statistics tracking to allow code reuse across both batch and streaming execution modes

2019-11-21 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-945. --- > Refactor Kafka extractor statistics tracking to allow code reuse across both > batch

[jira] [Closed] (GOBBLIN-955) Expose a method to get average record size in KafkaExtractorStatsTracker

2019-11-21 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-955. --- > Expose a method to get average record size in KafkaExtractorStatsTracker >

[jira] [Closed] (GOBBLIN-876) Expose metrics() API in GobblinKafkaConsumerClient to allow consume metrics to be reported

2019-11-21 Thread Sudarshan Vasudevan (Jira)
[ https://issues.apache.org/jira/browse/GOBBLIN-876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudarshan Vasudevan closed GOBBLIN-876. --- > Expose metrics() API in GobblinKafkaConsumerClient to allow consume metrics > to be

  1   2   3   >