[jira] [Created] (GOBBLIN-186) Add support for using the Kerberos authentication plugin without a GobblinDriverInstance
Hung Tran created GOBBLIN-186: - Summary: Add support for using the Kerberos authentication plugin without a GobblinDriverInstance Key: GOBBLIN-186 URL: https://issues.apache.org/jira/browse/GOBBLIN-186 Project: Apache Gobblin Issue Type: Bug Reporter: Hung Tran Instantiating the HadoopKerberosKeytabAuthenticationPlugin requires a GobblinInstanceDriver. There are instances where a GobblinInstanceDriver is not available. This plugin should be able to be instantiated with only configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GOBBLIN-186) Add support for using the Kerberos authentication plugin without a GobblinDriverInstance
[ https://issues.apache.org/jira/browse/GOBBLIN-186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran reassigned GOBBLIN-186: - Assignee: Hung Tran > Add support for using the Kerberos authentication plugin without a > GobblinDriverInstance > > > Key: GOBBLIN-186 > URL: https://issues.apache.org/jira/browse/GOBBLIN-186 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > Instantiating the HadoopKerberosKeytabAuthenticationPlugin requires a > GobblinInstanceDriver. There are instances where a GobblinInstanceDriver is > not available. This plugin should be able to be instantiated with only > configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (GOBBLIN-171) Add a writer wrapper that closes the wrapped writer and creates a new one
[ https://issues.apache.org/jira/browse/GOBBLIN-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran updated GOBBLIN-171: -- Sprint: Apache Gobblin 170724 > Add a writer wrapper that closes the wrapped writer and creates a new one > - > > Key: GOBBLIN-171 > URL: https://issues.apache.org/jira/browse/GOBBLIN-171 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Closing of a writer on flush is required to support intermediate publishing > of data to filesystems. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-171) Add a writer wrapper that closes the wrapped writer and creates a new one
Hung Tran created GOBBLIN-171: - Summary: Add a writer wrapper that closes the wrapped writer and creates a new one Key: GOBBLIN-171 URL: https://issues.apache.org/jira/browse/GOBBLIN-171 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Assignee: Hung Tran Closing of a writer on flush is required to support intermediate publishing of data to filesystems. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-191) Make sure cron scheduler works and tune schedule period
[ https://issues.apache.org/jira/browse/GOBBLIN-191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-191. --- Resolution: Fixed Issue resolved by pull request #2042 [https://github.com/apache/incubator-gobblin/pull/2042] > Make sure cron scheduler works and tune schedule period > --- > > Key: GOBBLIN-191 > URL: https://issues.apache.org/jira/browse/GOBBLIN-191 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Abhishek Tiwari >Assignee: Abhishek Tiwari > > Make sure cron scheduler works and tune schedule period. Right not it is not > copying over the required property from FlowSpec to the jobConfig in > ServiceScheduler. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-173) Add pattern support for job-level blacklist in distcpNG/replication
[ https://issues.apache.org/jira/browse/GOBBLIN-173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-173. --- Resolution: Fixed Issue resolved by pull request #2015 [https://github.com/apache/incubator-gobblin/pull/2015] > Add pattern support for job-level blacklist in distcpNG/replication > > > Key: GOBBLIN-173 > URL: https://issues.apache.org/jira/browse/GOBBLIN-173 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Assignee: Lei Sun > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-195) Ability to switch Avro schema namespace switch before registering with Kafka Avro Schema registry
[ https://issues.apache.org/jira/browse/GOBBLIN-195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-195. --- Resolution: Fixed Issue resolved by pull request #2049 [https://github.com/apache/incubator-gobblin/pull/2049] > Ability to switch Avro schema namespace switch before registering with Kafka > Avro Schema registry > - > > Key: GOBBLIN-195 > URL: https://issues.apache.org/jira/browse/GOBBLIN-195 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Abhishek Tiwari >Assignee: Abhishek Tiwari > > Ability to switch Avro schema namespace switch before registering with Kafka > Avro Schema registry. This is useful when we want to maintain backward > compatibility with the schema registered in the registry since registry does > not allows for difference in namespace. > This however has no impact on write / read of actual Avro bytes for the event > because they are mapped back to the fields irrespective of namespace. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-186) Add support for using the Kerberos authentication plugin without a GobblinDriverInstance
[ https://issues.apache.org/jira/browse/GOBBLIN-186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-186. --- Resolution: Fixed Issue resolved by pull request #2041 [https://github.com/apache/incubator-gobblin/pull/2041] > Add support for using the Kerberos authentication plugin without a > GobblinDriverInstance > > > Key: GOBBLIN-186 > URL: https://issues.apache.org/jira/browse/GOBBLIN-186 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > Instantiating the HadoopKerberosKeytabAuthenticationPlugin requires a > GobblinInstanceDriver. There are instances where a GobblinInstanceDriver is > not available. This plugin should be able to be instantiated with only > configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-184) Call the flush method of CloseOnFlushWriterWrapper when a FlushControlMessage is received
[ https://issues.apache.org/jira/browse/GOBBLIN-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-184. --- Resolution: Fixed Issue resolved by pull request #2040 [https://github.com/apache/incubator-gobblin/pull/2040] > Call the flush method of CloseOnFlushWriterWrapper when a FlushControlMessage > is received > - > > Key: GOBBLIN-184 > URL: https://issues.apache.org/jira/browse/GOBBLIN-184 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > When the close on flush functionality is enabled, the > CloseOnFlushWriterWrapper's flush method should be invoked when a > FlushControlMessage is received. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-204) Add a service that fetches GaaS flow configs from a git repository
[ https://issues.apache.org/jira/browse/GOBBLIN-204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-204. --- Resolution: Fixed Issue resolved by pull request #2055 [https://github.com/apache/incubator-gobblin/pull/2055] > Add a service that fetches GaaS flow configs from a git repository > -- > > Key: GOBBLIN-204 > URL: https://issues.apache.org/jira/browse/GOBBLIN-204 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > Add a service that polls a git repository for flow configuration. The service > is integrated with GaaS alongside the REST-based API for flow spec creation. > Both methods of flow management can co-exist, but should be used to manage > disjoint flows. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-215) hasJoinOperation failed when SQL statement has limit keyword
[ https://issues.apache.org/jira/browse/GOBBLIN-215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-215. --- Resolution: Fixed Issue resolved by pull request #2050 [https://github.com/apache/incubator-gobblin/pull/2050] > hasJoinOperation failed when SQL statement has limit keyword > > > Key: GOBBLIN-215 > URL: https://issues.apache.org/jira/browse/GOBBLIN-215 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-214) Filtering doesn't work in FileListUtils:listFilesRecursively
[ https://issues.apache.org/jira/browse/GOBBLIN-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-214. --- Resolution: Fixed Issue resolved by pull request #2067 [https://github.com/apache/incubator-gobblin/pull/2067] > Filtering doesn't work in FileListUtils:listFilesRecursively > > > Key: GOBBLIN-214 > URL: https://issues.apache.org/jira/browse/GOBBLIN-214 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu > > The filtering logic for FileListUtils:listFilesRecursively was wrong. It > never applies the filtering to the files that is non-directory type -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-238) Implement EnvelopePayloadExtractor and EnvelopePayloadDeserializer
[
https://issues.apache.org/jira/browse/GOBBLIN-238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hung Tran resolved GOBBLIN-238.
---
Resolution: Fixed
Issue resolved by pull request #2099
[https://github.com/apache/incubator-gobblin/pull/2099]
> Implement EnvelopePayloadExtractor and EnvelopePayloadDeserializer
> --
>
> Key: GOBBLIN-238
> URL: https://issues.apache.org/jira/browse/GOBBLIN-238
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
> Labels: Core:Converter
>
> h3. Why
> The current implementation of EnvelopeSchemaConverter has several flaws:
> - Assumes top level payload schema field
> - Output record is the schema'ed payload but output schema is a String
> To address the issues and improve envelope schema conversion, the task
> implements two types of EnvelopeSchemaConverter: EnvelopePayloadExtractor and
> EnvelopePayloadDeserializer.
> h3. EnvelopePayloadExtractor
> This is a replacement of the deprecated `EvenlopeSchemaConverter`. Given an
> envelope record, the output schema will be the latest payload schema fetched
> from a kafka registry. The output record will be the deserialized payload
> with the latest schema
> h3. EnvelopePayloadDeserializer
> Given an envelope record, the output schema will set the payload field to
> have the latest schema fetched from a kafka registry and set the other fields
> as they are from the input schema. The output record will set the payload to
> be the deserialized object with the latest schema and set the other fields as
> they are from the input record
> h3. Configurations
> One configuration is required to set for any of the converters to work. It
> has no default value.
> {code:java}
> // The topic to fetch the latest schema of the payload from a kafka registry
> converter.envelopeSchemaConverter.payloadSchemaTopic=
> {code}
> The converter supports nested schema id
> {code:java}
> converter.envelopeSchemaConverter.schemaIdField="metadata.payloadSchemaId"
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-256) Improve logging for gobblin compaction
[ https://issues.apache.org/jira/browse/GOBBLIN-256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-256. --- Resolution: Fixed Issue resolved by pull request #2108 [https://github.com/apache/incubator-gobblin/pull/2108] > Improve logging for gobblin compaction > -- > > Key: GOBBLIN-256 > URL: https://issues.apache.org/jira/browse/GOBBLIN-256 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Kuai Yu >Assignee: Kuai Yu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-234) Add a ControlMessageInjector that generates metadata update control messages
[ https://issues.apache.org/jira/browse/GOBBLIN-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-234. --- Resolution: Fixed Issue resolved by pull request #2107 [https://github.com/apache/incubator-gobblin/pull/2107] > Add a ControlMessageInjector that generates metadata update control messages > > > Key: GOBBLIN-234 > URL: https://issues.apache.org/jira/browse/GOBBLIN-234 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > Some converters use the latest registered schema at the time of converter > initialization. This schema may change and the converter chain may need to be > updated. A control message can be used to update the converters and notify > writers of schema changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-259) Support writing Kafka messages to db/table file path
[
https://issues.apache.org/jira/browse/GOBBLIN-259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hung Tran resolved GOBBLIN-259.
---
Resolution: Fixed
Issue resolved by pull request #2111
[https://github.com/apache/incubator-gobblin/pull/2111]
> Support writing Kafka messages to db/table file path
>
>
> Key: GOBBLIN-259
> URL: https://issues.apache.org/jira/browse/GOBBLIN-259
> Project: Apache Gobblin
> Issue Type: Bug
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
> Labels: Writer:HDFS
>
> - Add a new write file path type `DB_TABLE`, which writes records from an
> `Extract` to folder /.
> - A gobblin job which uses the `KafkSource` and `AvroHdfsDataWriter` will
> write records to 'dbName/tableName' with the following configurations:
> {code:java}
> extract.namespace=dbName
> extract.table.name=tableName
> writer.file.path.type=db_table
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-254) Add config key to update watermark when a partition is empty
[ https://issues.apache.org/jira/browse/GOBBLIN-254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-254. --- Resolution: Fixed Issue resolved by pull request #2105 [https://github.com/apache/incubator-gobblin/pull/2105] > Add config key to update watermark when a partition is empty > > > Key: GOBBLIN-254 > URL: https://issues.apache.org/jira/browse/GOBBLIN-254 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Assignee: Jack Moseley > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (GOBBLIN-234) Add a converter that generates metadata update control messages
[ https://issues.apache.org/jira/browse/GOBBLIN-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran updated GOBBLIN-234: -- Sprint: Apache Gobblin 170905 > Add a converter that generates metadata update control messages > --- > > Key: GOBBLIN-234 > URL: https://issues.apache.org/jira/browse/GOBBLIN-234 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > Some converters use the latest registered schema at the time of converter > initialization. This schema may change and the converter chain may need to be > updated. A control message can be used to update the converters and notify > writers of schema changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-236) Add a ControlMessage injector as a RecordStreamProcessor
[ https://issues.apache.org/jira/browse/GOBBLIN-236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-236. --- Resolution: Fixed Issue resolved by pull request #2090 [https://github.com/apache/incubator-gobblin/pull/2090] > Add a ControlMessage injector as a RecordStreamProcessor > > > Key: GOBBLIN-236 > URL: https://issues.apache.org/jira/browse/GOBBLIN-236 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Add a ControlMessage injector as a RecordStreamProcessor. > A ControlMessageInjector inspects an incoming record and can inject > ControlMessages based on the content of the incoming record. > One use case for this is the injection of MetadataUpdateControlMessages when > the latest schema has been updated to trigger the update of the schema used > by other constructs downstream from the ControlMessageInjector. > Long running jobs are more likely to encounter issues due to the schema being > updated after the start of the job. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-252) Add some azkaban related constants
[ https://issues.apache.org/jira/browse/GOBBLIN-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-252. --- Resolution: Fixed Issue resolved by pull request #2103 [https://github.com/apache/incubator-gobblin/pull/2103] > Add some azkaban related constants > -- > > Key: GOBBLIN-252 > URL: https://issues.apache.org/jira/browse/GOBBLIN-252 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Kuai Yu >Assignee: Kuai Yu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-264) Add a SharedResourceFactory for creating shared DataPublishers
Hung Tran created GOBBLIN-264: - Summary: Add a SharedResourceFactory for creating shared DataPublishers Key: GOBBLIN-264 URL: https://issues.apache.org/jira/browse/GOBBLIN-264 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Assignee: Hung Tran Reusable and sharable DataPublishers can reduce resource utilization. In streaming use cases a publisher may need to be invoked periodically to make data available. Reusing a publisher reduces initialization overhead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-258) Try to remove the tmp output path from wrong fs before compaction
[ https://issues.apache.org/jira/browse/GOBBLIN-258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-258. --- Resolution: Fixed Issue resolved by pull request #2110 [https://github.com/apache/incubator-gobblin/pull/2110] > Try to remove the tmp output path from wrong fs before compaction > - > > Key: GOBBLIN-258 > URL: https://issues.apache.org/jira/browse/GOBBLIN-258 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-compaction >Reporter: Tamas Nemeth >Assignee: Issac Buenrostro > > With my last pull request I introduced a bug as well which caused it tried to > remove the tmp output path from wrong fs before compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-274) Fix wait for salesforce batch completion
[ https://issues.apache.org/jira/browse/GOBBLIN-274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-274. --- Resolution: Fixed Issue resolved by pull request #2127 [https://github.com/apache/incubator-gobblin/pull/2127] > Fix wait for salesforce batch completion > > > Key: GOBBLIN-274 > URL: https://issues.apache.org/jira/browse/GOBBLIN-274 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > The wait for batch completion is broken when pk chunking is not enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-278) Fix sending lineage event for KafkaSource
[ https://issues.apache.org/jira/browse/GOBBLIN-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-278. --- Resolution: Fixed Issue resolved by pull request #2131 [https://github.com/apache/incubator-gobblin/pull/2131] > Fix sending lineage event for KafkaSource > - > > Key: GOBBLIN-278 > URL: https://issues.apache.org/jira/browse/GOBBLIN-278 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen > > 1. Fix lineage event for KafkaSource not send, and void resending the events > by removing configurations with key prefix `gobblin.lineage` from the state > 2. Fix `KafkaWorkUnitPacker` disregards existing configurations of work units -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-288) Add finer-grain dynamic partition generation for Salesforce
[ https://issues.apache.org/jira/browse/GOBBLIN-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-288. --- Resolution: Fixed Issue resolved by pull request #2140 [https://github.com/apache/incubator-gobblin/pull/2140] > Add finer-grain dynamic partition generation for Salesforce > --- > > Key: GOBBLIN-288 > URL: https://issues.apache.org/jira/browse/GOBBLIN-288 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-288) Add finer-grain dynamic partition generation for Salesforce
Hung Tran created GOBBLIN-288: - Summary: Add finer-grain dynamic partition generation for Salesforce Key: GOBBLIN-288 URL: https://issues.apache.org/jira/browse/GOBBLIN-288 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Assignee: Hung Tran -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-244) Need additional info for gobblin tracking hourly-deduped
[ https://issues.apache.org/jira/browse/GOBBLIN-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-244. --- Resolution: Fixed Issue resolved by pull request #2094 [https://github.com/apache/incubator-gobblin/pull/2094] > Need additional info for gobblin tracking hourly-deduped > > > Key: GOBBLIN-244 > URL: https://issues.apache.org/jira/browse/GOBBLIN-244 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu > > Add the previous record count and the number of execution runs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-245) Create topic specific extract of a WorkUnit in KafkaSource
[ https://issues.apache.org/jira/browse/GOBBLIN-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-245. --- Resolution: Fixed Issue resolved by pull request #2095 [https://github.com/apache/incubator-gobblin/pull/2095] > Create topic specific extract of a WorkUnit in KafkaSource > -- > > Key: GOBBLIN-245 > URL: https://issues.apache.org/jira/browse/GOBBLIN-245 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen > Labels: Source:Kafka > > Current KafkaSource ignores topic specific configurations on creating Extract > of a WorkUnit. The task is to create the extract with topic specific > configurations if any or else job level configurations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (GOBBLIN-264) Add a SharedResourceFactory for creating shared DataPublishers
[ https://issues.apache.org/jira/browse/GOBBLIN-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran updated GOBBLIN-264: -- Sprint: Apache Gobblin 170905 > Add a SharedResourceFactory for creating shared DataPublishers > -- > > Key: GOBBLIN-264 > URL: https://issues.apache.org/jira/browse/GOBBLIN-264 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Reusable and sharable DataPublishers can reduce resource utilization. In > streaming use cases a publisher may need to be invoked periodically to make > data available. Reusing a publisher reduces initialization overhead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-265) Add support for PK chunking to gobblin-salesforce
Hung Tran created GOBBLIN-265: - Summary: Add support for PK chunking to gobblin-salesforce Key: GOBBLIN-265 URL: https://issues.apache.org/jira/browse/GOBBLIN-265 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Some data sets have modification time clustering that results in query timeout due to too many rows being fetched in a bulk API call. Add support for enabling PK chunking to avoid the timeout error. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-260) Salesforce dynamic partitioning bugs
[ https://issues.apache.org/jira/browse/GOBBLIN-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-260. --- Resolution: Fixed Issue resolved by pull request #2112 [https://github.com/apache/incubator-gobblin/pull/2112] > Salesforce dynamic partitioning bugs > > > Key: GOBBLIN-260 > URL: https://issues.apache.org/jira/browse/GOBBLIN-260 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Assignee: Jack Moseley > > 1. When source.max.number.of.partitions = 1 and dynamic partitioning is > enabled, no data is output. > 2. When dynamic partitioning is enabled, incremental run ignores low > watermark and pulls from the start of the year. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-268) Unique job uri and job name generation for GaaS
[ https://issues.apache.org/jira/browse/GOBBLIN-268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-268. --- Resolution: Fixed Issue resolved by pull request #2121 [https://github.com/apache/incubator-gobblin/pull/2121] > Unique job uri and job name generation for GaaS > --- > > Key: GOBBLIN-268 > URL: https://issues.apache.org/jira/browse/GOBBLIN-268 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-264) Add a SharedResourceFactory for creating shared DataPublishers
[ https://issues.apache.org/jira/browse/GOBBLIN-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-264. --- Resolution: Fixed Issue resolved by pull request #2116 [https://github.com/apache/incubator-gobblin/pull/2116] > Add a SharedResourceFactory for creating shared DataPublishers > -- > > Key: GOBBLIN-264 > URL: https://issues.apache.org/jira/browse/GOBBLIN-264 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Reusable and sharable DataPublishers can reduce resource utilization. In > streaming use cases a publisher may need to be invoked periodically to make > data available. Reusing a publisher reduces initialization overhead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (GOBBLIN-271) Move the grok converter to the gobblin-grok module
[ https://issues.apache.org/jira/browse/GOBBLIN-271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran updated GOBBLIN-271: -- Sprint: Apache Gobblin 170905 > Move the grok converter to the gobblin-grok module > -- > > Key: GOBBLIN-271 > URL: https://issues.apache.org/jira/browse/GOBBLIN-271 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-271) Move the grok converter to the gobblin-grok module
[ https://issues.apache.org/jira/browse/GOBBLIN-271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-271. --- Resolution: Fixed Issue resolved by pull request #2123 [https://github.com/apache/incubator-gobblin/pull/2123] > Move the grok converter to the gobblin-grok module > -- > > Key: GOBBLIN-271 > URL: https://issues.apache.org/jira/browse/GOBBLIN-271 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GOBBLIN-265) Add support for PK chunking to gobblin-salesforce
[ https://issues.apache.org/jira/browse/GOBBLIN-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran reassigned GOBBLIN-265: - Assignee: Hung Tran > Add support for PK chunking to gobblin-salesforce > - > > Key: GOBBLIN-265 > URL: https://issues.apache.org/jira/browse/GOBBLIN-265 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Some data sets have modification time clustering that results in query > timeout due to too many rows being fetched in a bulk API call. Add support > for enabling PK chunking to avoid the timeout error. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-267) HiveSource creates workunit even when update time is before maxLookBackDays
[ https://issues.apache.org/jira/browse/GOBBLIN-267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-267. --- Resolution: Fixed Issue resolved by pull request #2119 [https://github.com/apache/incubator-gobblin/pull/2119] > HiveSource creates workunit even when update time is before maxLookBackDays > --- > > Key: GOBBLIN-267 > URL: https://issues.apache.org/jira/browse/GOBBLIN-267 > Project: Apache Gobblin > Issue Type: Bug > Components: misc >Reporter: Aditya Sharma >Assignee: Aditya Sharma > > org.apache.gobblin.data.management.conversion.hive.source.HiveSource creates > workunit if: > 1) Create time is after maxLookBackDays > 2) Update time is greater than watermark > Since there are multiple policies to decide update time, it can happen that > create time is greater than update time and hence maxLookBackDays will be > redundant. > HiveSource should check for maxLookBackDays corresponding to update time -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GOBBLIN-225) Fix cloning of ControlMessages in PartitionDataWriterMessageHandler
[ https://issues.apache.org/jira/browse/GOBBLIN-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran reassigned GOBBLIN-225: - Assignee: Hung Tran > Fix cloning of ControlMessages in PartitionDataWriterMessageHandler > --- > > Key: GOBBLIN-225 > URL: https://issues.apache.org/jira/browse/GOBBLIN-225 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > PartitionDataWriterMessageHandler does a single clone even though there can > be multiple partitions that needs to handle the ControlMessage. This class > needs to use a fork clone to create messages to pass to the partitioned > handlers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-225) Fix cloning of ControlMessages in PartitionDataWriterMessageHandler
Hung Tran created GOBBLIN-225: - Summary: Fix cloning of ControlMessages in PartitionDataWriterMessageHandler Key: GOBBLIN-225 URL: https://issues.apache.org/jira/browse/GOBBLIN-225 Project: Apache Gobblin Issue Type: Bug Reporter: Hung Tran PartitionDataWriterMessageHandler does a single clone even though there can be multiple partitions that needs to handle the ControlMessage. This class needs to use a fork clone to create messages to pass to the partitioned handlers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-212) Exception handling of TaskStateCollectorServiceHandler
[ https://issues.apache.org/jira/browse/GOBBLIN-212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-212. --- Resolution: Fixed Issue resolved by pull request #2064 [https://github.com/apache/incubator-gobblin/pull/2064] > Exception handling of TaskStateCollectorServiceHandler > -- > > Key: GOBBLIN-212 > URL: https://issues.apache.org/jira/browse/GOBBLIN-212 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Lei Sun >Assignee: Lei Sun > > Current if the TaskStateCollectorServiceHandler failed, the whole job won't > proceed. It is not the correct behavior sometimes if we would like the job to > proceed and finish dataset commit even there's something wrong happen in the > TaskStateCollectorServiceHandler. Should catch the exception carefully and > make it configurable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-32) StateStores created with rootDir that is incompatible with state.store.type
[ https://issues.apache.org/jira/browse/GOBBLIN-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-32. -- Resolution: Fixed > StateStores created with rootDir that is incompatible with state.store.type > --- > > Key: GOBBLIN-32 > URL: https://issues.apache.org/jira/browse/GOBBLIN-32 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Joel Baranick >Assignee: Hung Tran > > The StateStores class, when run under gobblin-yarn, can be created with a > rootDir (which comes from the yarn application work directory and is in the > form of `HDFS://...`) that is incompatible with the configured > `state.store.type`. > > *Github Url* : https://github.com/linkedin/gobblin/issues/1848 > *Github Reporter* : [~jbaranick] > *Github Created At* : 2017-05-09T17:30:00Z > *Github Updated At* : 2017-06-22T21:36:54Z > h3. Comments > > [~jbaranick] wrote on 2017-06-22T21:36:54Z : @htran1 Are you able to look > into this? > > *Github Url* : > https://github.com/linkedin/gobblin/issues/1848#issuecomment-310509726 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GOBBLIN-32) StateStores created with rootDir that is incompatible with state.store.type
[ https://issues.apache.org/jira/browse/GOBBLIN-32?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135430#comment-16135430 ] Hung Tran commented on GOBBLIN-32: -- PR https://github.com/apache/incubator-gobblin/pull/2035 has been merged. > StateStores created with rootDir that is incompatible with state.store.type > --- > > Key: GOBBLIN-32 > URL: https://issues.apache.org/jira/browse/GOBBLIN-32 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Joel Baranick >Assignee: Hung Tran > > The StateStores class, when run under gobblin-yarn, can be created with a > rootDir (which comes from the yarn application work directory and is in the > form of `HDFS://...`) that is incompatible with the configured > `state.store.type`. > > *Github Url* : https://github.com/linkedin/gobblin/issues/1848 > *Github Reporter* : [~jbaranick] > *Github Created At* : 2017-05-09T17:30:00Z > *Github Updated At* : 2017-06-22T21:36:54Z > h3. Comments > > [~jbaranick] wrote on 2017-06-22T21:36:54Z : @htran1 Are you able to look > into this? > > *Github Url* : > https://github.com/linkedin/gobblin/issues/1848#issuecomment-310509726 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-235) Prevent log warnings when TaskStateCollectorService has no task states detected
[ https://issues.apache.org/jira/browse/GOBBLIN-235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-235. --- Resolution: Fixed Issue resolved by pull request #2087 [https://github.com/apache/incubator-gobblin/pull/2087] > Prevent log warnings when TaskStateCollectorService has no task states > detected > --- > > Key: GOBBLIN-235 > URL: https://issues.apache.org/jira/browse/GOBBLIN-235 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu > > Need to adjust log level from warning to debug -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GOBBLIN-234) Add a converter that generates metadata update control messages
[ https://issues.apache.org/jira/browse/GOBBLIN-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran reassigned GOBBLIN-234: - Assignee: Hung Tran > Add a converter that generates metadata update control messages > --- > > Key: GOBBLIN-234 > URL: https://issues.apache.org/jira/browse/GOBBLIN-234 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > Some converters use the latest registered schema at the time of converter > initialization. This schema may change and the converter chain may need to be > updated. A control message can be used to update the converters and notify > writers of schema changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-276) Change setActive order to prevent flow spec loss
[ https://issues.apache.org/jira/browse/GOBBLIN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-276. --- Resolution: Fixed Issue resolved by pull request #2129 [https://github.com/apache/incubator-gobblin/pull/2129] > Change setActive order to prevent flow spec loss > > > Key: GOBBLIN-276 > URL: https://issues.apache.org/jira/browse/GOBBLIN-276 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu > > 1. Originally isActive=true is set after onAddSpec was invoked during the > leadership change. However this has a problem because onAddSpec will forward > a spec to new leader when isActive==false, but actually current node is > already the new leader. > 2. Put some atomic boolean variable to protect the topology being initialized > before we load flow specs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-284) Add retry in SalesforceExtractor to handle transient network errors
[ https://issues.apache.org/jira/browse/GOBBLIN-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-284. --- Resolution: Fixed Issue resolved by pull request #2137 [https://github.com/apache/incubator-gobblin/pull/2137] > Add retry in SalesforceExtractor to handle transient network errors > --- > > Key: GOBBLIN-284 > URL: https://issues.apache.org/jira/browse/GOBBLIN-284 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-283) Refactor EnvelopePayloadConverter to support multi fields conversion
[ https://issues.apache.org/jira/browse/GOBBLIN-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-283. --- Resolution: Fixed Issue resolved by pull request #2136 [https://github.com/apache/incubator-gobblin/pull/2136] > Refactor EnvelopePayloadConverter to support multi fields conversion > > > Key: GOBBLIN-283 > URL: https://issues.apache.org/jira/browse/GOBBLIN-283 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GOBBLIN-284) Add retry in SalesforceExtractor to handle transient network errors
[ https://issues.apache.org/jira/browse/GOBBLIN-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran reassigned GOBBLIN-284: - Assignee: Hung Tran > Add retry in SalesforceExtractor to handle transient network errors > --- > > Key: GOBBLIN-284 > URL: https://issues.apache.org/jira/browse/GOBBLIN-284 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-284) Add retry in SalesforceExtractor to handle transient network errors
Hung Tran created GOBBLIN-284: - Summary: Add retry in SalesforceExtractor to handle transient network errors Key: GOBBLIN-284 URL: https://issues.apache.org/jira/browse/GOBBLIN-284 Project: Apache Gobblin Issue Type: Bug Reporter: Hung Tran -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (GOBBLIN-284) Add retry in SalesforceExtractor to handle transient network errors
[ https://issues.apache.org/jira/browse/GOBBLIN-284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran updated GOBBLIN-284: -- Sprint: Apache Gobblin 170905 > Add retry in SalesforceExtractor to handle transient network errors > --- > > Key: GOBBLIN-284 > URL: https://issues.apache.org/jira/browse/GOBBLIN-284 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-325) Add a Source and Extractor for stress testing
Hung Tran created GOBBLIN-325: - Summary: Add a Source and Extractor for stress testing Key: GOBBLIN-325 URL: https://issues.apache.org/jira/browse/GOBBLIN-325 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Assignee: Hung Tran Add a Source and Extractor that has the following functionality. * Configurable sleep time per record * Configurable compute time per record * Run duration or record count limit per extractor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-325) Add a Source and Extractor for stress testing
[ https://issues.apache.org/jira/browse/GOBBLIN-325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-325. --- Resolution: Fixed Issue resolved by pull request #2177 [https://github.com/apache/incubator-gobblin/pull/2177] > Add a Source and Extractor for stress testing > - > > Key: GOBBLIN-325 > URL: https://issues.apache.org/jira/browse/GOBBLIN-325 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Add a Source and Extractor that has the following functionality. > * Configurable sleep time per record > * Configurable compute time per record > * Run duration or record count limit per extractor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-331) Add sharedConfig support for the KafkaDataWriters
Hung Tran created GOBBLIN-331: - Summary: Add sharedConfig support for the KafkaDataWriters Key: GOBBLIN-331 URL: https://issues.apache.org/jira/browse/GOBBLIN-331 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Shared kafka configuration is currently not passed to the KafkaDataWriter. There are some commonly shared config, such as SSL keystore location that should be used to configure the KafkaDataWriter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-328) GobblinClusterKillTest failed. Not able to find expected output files.
[ https://issues.apache.org/jira/browse/GOBBLIN-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-328. --- Resolution: Fixed Issue resolved by pull request #2180 [https://github.com/apache/incubator-gobblin/pull/2180] > GobblinClusterKillTest failed. Not able to find expected output files. > -- > > Key: GOBBLIN-328 > URL: https://issues.apache.org/jira/browse/GOBBLIN-328 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-cluster >Reporter: Ray Yang >Assignee: Hung Tran > > Issue: > org.apache.gobblin.cluster.GobblinClusterKillTest failed because it looks at > the wrong output path for the output files. > Cause: > It appears that the paths have changed. > Fix: > Update the path to match. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GOBBLIN-331) Add sharedConfig support for the KafkaDataWriters
[ https://issues.apache.org/jira/browse/GOBBLIN-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran reassigned GOBBLIN-331: - Assignee: Hung Tran > Add sharedConfig support for the KafkaDataWriters > - > > Key: GOBBLIN-331 > URL: https://issues.apache.org/jira/browse/GOBBLIN-331 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Shared kafka configuration is currently not passed to the KafkaDataWriter. > There are some commonly shared config, such as SSL keystore location that > should be used to configure the KafkaDataWriter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-331) Add sharedConfig support for the KafkaDataWriters
[ https://issues.apache.org/jira/browse/GOBBLIN-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-331. --- Resolution: Fixed Issue resolved by pull request #2183 [https://github.com/apache/incubator-gobblin/pull/2183] > Add sharedConfig support for the KafkaDataWriters > - > > Key: GOBBLIN-331 > URL: https://issues.apache.org/jira/browse/GOBBLIN-331 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran > > Shared kafka configuration is currently not passed to the KafkaDataWriter. > There are some commonly shared config, such as SSL keystore location that > should be used to configure the KafkaDataWriter. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-329) Add a basic cluster integration test
[ https://issues.apache.org/jira/browse/GOBBLIN-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-329. --- Resolution: Fixed Issue resolved by pull request #2181 [https://github.com/apache/incubator-gobblin/pull/2181] > Add a basic cluster integration test > > > Key: GOBBLIN-329 > URL: https://issues.apache.org/jira/browse/GOBBLIN-329 > Project: Apache Gobblin > Issue Type: Test > Components: gobblin-cluster >Reporter: Ray Yang >Assignee: Hung Tran > > Add a new basic cluster integration test > This is useful to verify that the basic function is working. > It also helps new engineers an easy way to learn the code. > The closest tests I can find so far is GobblinClusterKillTest. But that is > testing some error handling and is slow and disabled due to reliability > issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-346) Close KafkaPusher in the KafkaEventReporter
Hung Tran created GOBBLIN-346: - Summary: Close KafkaPusher in the KafkaEventReporter Key: GOBBLIN-346 URL: https://issues.apache.org/jira/browse/GOBBLIN-346 Project: Apache Gobblin Issue Type: Bug Reporter: Hung Tran Assignee: Hung Tran -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-347) KafkaPusher is not closed when GobblinMetrics.stopReporting is called
[ https://issues.apache.org/jira/browse/GOBBLIN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-347. --- Resolution: Fixed Issue resolved by pull request #2206 [https://github.com/apache/incubator-gobblin/pull/2206] > KafkaPusher is not closed when GobblinMetrics.stopReporting is called > - > > Key: GOBBLIN-347 > URL: https://issues.apache.org/jira/browse/GOBBLIN-347 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-metrics >Reporter: Sunitha Beeram >Assignee: Issac Buenrostro > > KafkaPusher is not closed when GobblinMetrics.stopReporting is called, > resulting in queued messages not getting sent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-317) Add dynamic configuration injection in the mappers
[ https://issues.apache.org/jira/browse/GOBBLIN-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-317. --- Resolution: Fixed Issue resolved by pull request #2170 [https://github.com/apache/incubator-gobblin/pull/2170] > Add dynamic configuration injection in the mappers > -- > > Key: GOBBLIN-317 > URL: https://issues.apache.org/jira/browse/GOBBLIN-317 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Some config, like SSL certificates may be distributed dynamically on mappers. > A way to generate and inject the dynamic config into the mappers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-317) Add dynamic configuration injection in the mappers
Hung Tran created GOBBLIN-317: - Summary: Add dynamic configuration injection in the mappers Key: GOBBLIN-317 URL: https://issues.apache.org/jira/browse/GOBBLIN-317 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Assignee: Hung Tran Some config, like SSL certificates may be distributed dynamically on mappers. A way to generate and inject the dynamic config into the mappers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-301) Fix the key GOBBLIN_KAFKA_CONSUMER_CLIENT_FACTORY_CLASS
Hung Tran created GOBBLIN-301: - Summary: Fix the key GOBBLIN_KAFKA_CONSUMER_CLIENT_FACTORY_CLASS Key: GOBBLIN-301 URL: https://issues.apache.org/jira/browse/GOBBLIN-301 Project: Apache Gobblin Issue Type: Bug Reporter: Hung Tran Assignee: Hung Tran The value of GOBBLIN_KAFKA_CONSUMER_CLIENT_FACTORY_CLASS in KafkaSource incorrectly has org.apache appended. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-301) Fix the key GOBBLIN_KAFKA_CONSUMER_CLIENT_FACTORY_CLASS
[ https://issues.apache.org/jira/browse/GOBBLIN-301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-301. --- Resolution: Fixed Issue resolved by pull request #2156 [https://github.com/apache/incubator-gobblin/pull/2156] > Fix the key GOBBLIN_KAFKA_CONSUMER_CLIENT_FACTORY_CLASS > --- > > Key: GOBBLIN-301 > URL: https://issues.apache.org/jira/browse/GOBBLIN-301 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > The value of GOBBLIN_KAFKA_CONSUMER_CLIENT_FACTORY_CLASS in KafkaSource > incorrectly has org.apache appended. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-303) Compaction can generate zero sized output when MR is in speculative mode
[ https://issues.apache.org/jira/browse/GOBBLIN-303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-303. --- Resolution: Fixed Issue resolved by pull request #2158 [https://github.com/apache/incubator-gobblin/pull/2158] > Compaction can generate zero sized output when MR is in speculative mode > > > Key: GOBBLIN-303 > URL: https://issues.apache.org/jira/browse/GOBBLIN-303 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu >Priority: Minor > > Currently if MR job used speculative mode, it was very likely that output has > a zero sized file generated by a killed task attempt. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-308) Gobblin cluster bootup hangs
[ https://issues.apache.org/jira/browse/GOBBLIN-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-308. --- Resolution: Fixed Issue resolved by pull request #2162 [https://github.com/apache/incubator-gobblin/pull/2162] > Gobblin cluster bootup hangs > > > Key: GOBBLIN-308 > URL: https://issues.apache.org/jira/browse/GOBBLIN-308 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu > > The problem happens when there are more than 100 files in the job catalog. > During the boot up sequence, spec consumer was launched after jobCatalog. > However the jobCatalog launches with a job listener which will push job spec > into a blocking queue, and due to spec consumer hasn't been started, no > component will start to consume job specs from the blocking queue. Once the > blocking queue max size (100 by default) is reached, the system is hanging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (GOBBLIN-310) Skip rerunning completed tasks on mapper reattempts
[ https://issues.apache.org/jira/browse/GOBBLIN-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran updated GOBBLIN-310: -- Sprint: Apache Gobblin 170905 > Skip rerunning completed tasks on mapper reattempts > --- > > Key: GOBBLIN-310 > URL: https://issues.apache.org/jira/browse/GOBBLIN-310 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > Subsequent executions of a failed mapper will rerun completed tasks. This can > result in duplicate data or errors due to collisions when publishing. > The state of completed mappers should be recorded and completed mappers > should be skipped on subsequent attemps. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GOBBLIN-310) Skip rerunning completed tasks on mapper reattempts
[ https://issues.apache.org/jira/browse/GOBBLIN-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran reassigned GOBBLIN-310: - Assignee: Hung Tran > Skip rerunning completed tasks on mapper reattempts > --- > > Key: GOBBLIN-310 > URL: https://issues.apache.org/jira/browse/GOBBLIN-310 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Hung Tran >Assignee: Hung Tran > > Subsequent executions of a failed mapper will rerun completed tasks. This can > result in duplicate data or errors due to collisions when publishing. > The state of completed mappers should be recorded and completed mappers > should be skipped on subsequent attemps. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (GOBBLIN-312) Pass extra kafka configuration to the KafkaConsumer in KafkaSimpleStreamingSource
[ https://issues.apache.org/jira/browse/GOBBLIN-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran reassigned GOBBLIN-312: - Assignee: Hung Tran > Pass extra kafka configuration to the KafkaConsumer in > KafkaSimpleStreamingSource > - > > Key: GOBBLIN-312 > URL: https://issues.apache.org/jira/browse/GOBBLIN-312 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Pass extra configuration to the KafkaConsumer. One use case is SSL > configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-312) Pass extra kafka configuration to the KafkaConsumer in KafkaSimpleStreamingSource
Hung Tran created GOBBLIN-312: - Summary: Pass extra kafka configuration to the KafkaConsumer in KafkaSimpleStreamingSource Key: GOBBLIN-312 URL: https://issues.apache.org/jira/browse/GOBBLIN-312 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Pass extra configuration to the KafkaConsumer. One use case is SSL configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (GOBBLIN-312) Pass extra kafka configuration to the KafkaConsumer in KafkaSimpleStreamingSource
[ https://issues.apache.org/jira/browse/GOBBLIN-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran updated GOBBLIN-312: -- Sprint: Apache Gobblin 170905 > Pass extra kafka configuration to the KafkaConsumer in > KafkaSimpleStreamingSource > - > > Key: GOBBLIN-312 > URL: https://issues.apache.org/jira/browse/GOBBLIN-312 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Pass extra configuration to the KafkaConsumer. One use case is SSL > configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-312) Pass extra kafka configuration to the KafkaConsumer in KafkaSimpleStreamingSource
[ https://issues.apache.org/jira/browse/GOBBLIN-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-312. --- Resolution: Fixed Issue resolved by pull request #2166 [https://github.com/apache/incubator-gobblin/pull/2166] > Pass extra kafka configuration to the KafkaConsumer in > KafkaSimpleStreamingSource > - > > Key: GOBBLIN-312 > URL: https://issues.apache.org/jira/browse/GOBBLIN-312 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > Pass extra configuration to the KafkaConsumer. One use case is SSL > configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-305) Add csv-kafka and kafka-hdfs template
[ https://issues.apache.org/jira/browse/GOBBLIN-305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-305. --- Resolution: Fixed Issue resolved by pull request #2160 [https://github.com/apache/incubator-gobblin/pull/2160] > Add csv-kafka and kafka-hdfs template > - > > Key: GOBBLIN-305 > URL: https://issues.apache.org/jira/browse/GOBBLIN-305 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen > > - Add 2 job templates: csv to kafka and kafka to hdfs > - Add type transformation in CsvToJsonConverterV2 converter -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-326) Gobblin metrics constructor only provides default constructor for Codhale metrics
[ https://issues.apache.org/jira/browse/GOBBLIN-326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-326. --- Resolution: Fixed Issue resolved by pull request #2178 [https://github.com/apache/incubator-gobblin/pull/2178] > Gobblin metrics constructor only provides default constructor for Codhale > metrics > - > > Key: GOBBLIN-326 > URL: https://issues.apache.org/jira/browse/GOBBLIN-326 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Kuai Yu >Assignee: Kuai Yu > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-333) Remove reference to log4j in WriterUtils
[ https://issues.apache.org/jira/browse/GOBBLIN-333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-333. --- Resolution: Fixed Issue resolved by pull request #2186 [https://github.com/apache/incubator-gobblin/pull/2186] > Remove reference to log4j in WriterUtils > > > Key: GOBBLIN-333 > URL: https://issues.apache.org/jira/browse/GOBBLIN-333 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > WriterUtils creates a log4j logger that is not used. This results in an > unnecessary dependency. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-335) Increase blob size in MySQL state store
[ https://issues.apache.org/jira/browse/GOBBLIN-335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-335. --- Resolution: Fixed Issue resolved by pull request #2189 [https://github.com/apache/incubator-gobblin/pull/2189] > Increase blob size in MySQL state store > --- > > Key: GOBBLIN-335 > URL: https://issues.apache.org/jira/browse/GOBBLIN-335 > Project: Apache Gobblin > Issue Type: Task >Reporter: Lei Sun >Assignee: Lei Sun > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-344) Fix help method getResolver in LineageInfo is private
[ https://issues.apache.org/jira/browse/GOBBLIN-344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-344. --- Resolution: Fixed Issue resolved by pull request #2200 [https://github.com/apache/incubator-gobblin/pull/2200] > Fix help method getResolver in LineageInfo is private > - > > Key: GOBBLIN-344 > URL: https://issues.apache.org/jira/browse/GOBBLIN-344 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen > > In the PR https://github.com/apache/incubator-gobblin/pull/2187, I mistakenly > made help method `LineageInfo#getResolver` private. It should be `public`. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-336) Gobblin Cluster Job Isolation
[ https://issues.apache.org/jira/browse/GOBBLIN-336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-336. --- Resolution: Fixed Issue resolved by pull request #2193 [https://github.com/apache/incubator-gobblin/pull/2193] > Gobblin Cluster Job Isolation > - > > Key: GOBBLIN-336 > URL: https://issues.apache.org/jira/browse/GOBBLIN-336 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Ray Yang > > Gobblin cluster runs Gobblin jobs. Each cluster worker host runs jobs in a > thread pool in a single JVM. The thread pool is reused for next jobs after > previous jobs finish. > Gobblin cluster recently ran into issues with resource leakage. The cluster > would fail all job executions when certain resources such as threads were > exhausted. To recover, the whole cluster has to be restarted and jobs have to > be retried. With the expected increase in the number of jobs executed, such > errors happen more frequently. We have identified the causes and fixes have > been verfied. However, there are concerns that unknown similar bugs may show > up later that may bring the whole cluster down. > In general, any bug in one job’s code may affect the executions of another > job since they run in the same JVM. It’s also possible that a bug will only > be triggered by certain input data which is specific to a subset of jobs. > The cluster will be more robust if a job execution is better isolated from > another job. > In the future, we expect jobs will become more diverse as more use cases are > on-boarded. The need for job isolation will become more important over time. > In the future job isolation may be required for security reasons too. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (GOBBLIN-333) Remove reference to log4j in WriterUtils
[ https://issues.apache.org/jira/browse/GOBBLIN-333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran updated GOBBLIN-333: -- Sprint: Apache Gobblin 170905 > Remove reference to log4j in WriterUtils > > > Key: GOBBLIN-333 > URL: https://issues.apache.org/jira/browse/GOBBLIN-333 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > WriterUtils creates a log4j logger that is not used. This results in an > unnecessary dependency. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-333) Remove reference to log4j in WriterUtils
Hung Tran created GOBBLIN-333: - Summary: Remove reference to log4j in WriterUtils Key: GOBBLIN-333 URL: https://issues.apache.org/jira/browse/GOBBLIN-333 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Assignee: Hung Tran WriterUtils creates a log4j logger that is not used. This results in an unnecessary dependency. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-286) Fix bug where non hive dataset throw NPE during dataset publish
[ https://issues.apache.org/jira/browse/GOBBLIN-286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-286. --- Resolution: Fixed Issue resolved by pull request #2148 [https://github.com/apache/incubator-gobblin/pull/2148] > Fix bug where non hive dataset throw NPE during dataset publish > --- > > Key: GOBBLIN-286 > URL: https://issues.apache.org/jira/browse/GOBBLIN-286 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Arjun Singh Bora >Assignee: Arjun Singh Bora > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-295) Make missing nullable fields default to null in json to avro converter
[ https://issues.apache.org/jira/browse/GOBBLIN-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-295. --- Resolution: Fixed Issue resolved by pull request #2146 [https://github.com/apache/incubator-gobblin/pull/2146] > Make missing nullable fields default to null in json to avro converter > -- > > Key: GOBBLIN-295 > URL: https://issues.apache.org/jira/browse/GOBBLIN-295 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Assignee: Jack Moseley > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (GOBBLIN-300) Use 1.7.7 form of Schema.createUnion() API that takes in a list
Hung Tran created GOBBLIN-300: - Summary: Use 1.7.7 form of Schema.createUnion() API that takes in a list Key: GOBBLIN-300 URL: https://issues.apache.org/jira/browse/GOBBLIN-300 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Assignee: Hung Tran 1.8.0 of avro introduced a form of Schema.createUnion() that takes in a variable length arguments. This API does not exist in 1.7.7 and causes problems for users who use the JsonElementConversionFactory with 1.7.7. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-300) Use 1.7.7 form of Schema.createUnion() API that takes in a list
[ https://issues.apache.org/jira/browse/GOBBLIN-300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-300. --- Resolution: Fixed Issue resolved by pull request #2155 [https://github.com/apache/incubator-gobblin/pull/2155] > Use 1.7.7 form of Schema.createUnion() API that takes in a list > --- > > Key: GOBBLIN-300 > URL: https://issues.apache.org/jira/browse/GOBBLIN-300 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran > > 1.8.0 of avro introduced a form of Schema.createUnion() that takes in a > variable length arguments. This API does not exist in 1.7.7 and causes > problems for users who use the JsonElementConversionFactory with 1.7.7. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (GOBBLIN-501) Fix NPE thrown from read after EOF of LazyMaterializeDecryptorInputStream
[ https://issues.apache.org/jira/browse/GOBBLIN-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-501. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2371 [https://github.com/apache/incubator-gobblin/pull/2371] > Fix NPE thrown from read after EOF of LazyMaterializeDecryptorInputStream > - > > Key: GOBBLIN-501 > URL: https://issues.apache.org/jira/browse/GOBBLIN-501 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen >Priority: Major > Fix For: 0.13.0 > > > A `read` call to a LazyMaterializeDecryptorInputStream when it reaches EOF > will throw a NPE. The fix is to return `-1` for any read after EOF. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-502) Make HiveMetastoreClient PoolCache's TTL configurable
[ https://issues.apache.org/jira/browse/GOBBLIN-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-502. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2372 [https://github.com/apache/incubator-gobblin/pull/2372] > Make HiveMetastoreClient PoolCache's TTL configurable > - > > Key: GOBBLIN-502 > URL: https://issues.apache.org/jira/browse/GOBBLIN-502 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Assignee: Lei Sun >Priority: Major > Fix For: 0.13.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-503) ForkThrowableHolder doesn't aggregate throwable in right condition
[ https://issues.apache.org/jira/browse/GOBBLIN-503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-503. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2373 [https://github.com/apache/incubator-gobblin/pull/2373] > ForkThrowableHolder doesn't aggregate throwable in right condition > -- > > Key: GOBBLIN-503 > URL: https://issues.apache.org/jira/browse/GOBBLIN-503 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Priority: Major > Fix For: 0.13.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-504) HiveMetastoreClientPool has findbugsMain issue due to unprotected static variable initialization
[ https://issues.apache.org/jira/browse/GOBBLIN-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-504. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2374 [https://github.com/apache/incubator-gobblin/pull/2374] > HiveMetastoreClientPool has findbugsMain issue due to unprotected static > variable initialization > > > Key: GOBBLIN-504 > URL: https://issues.apache.org/jira/browse/GOBBLIN-504 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu >Priority: Major > Fix For: 0.13.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-496) Support nullable unions in AvroUtils.getFieldSchema
[ https://issues.apache.org/jira/browse/GOBBLIN-496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-496. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2367 [https://github.com/apache/incubator-gobblin/pull/2367] > Support nullable unions in AvroUtils.getFieldSchema > --- > > Key: GOBBLIN-496 > URL: https://issues.apache.org/jira/browse/GOBBLIN-496 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Assignee: Jack Moseley >Priority: Major > Fix For: 0.13.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-495) FlowSpec should be deleted if this is run once flow
[ https://issues.apache.org/jira/browse/GOBBLIN-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-495. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2366 [https://github.com/apache/incubator-gobblin/pull/2366] > FlowSpec should be deleted if this is run once flow > --- > > Key: GOBBLIN-495 > URL: https://issues.apache.org/jira/browse/GOBBLIN-495 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Kuai Yu >Assignee: Kuai Yu >Priority: Major > Fix For: 0.13.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-492) Make LoopingDatasetFinderSource easy to embed different Iterator
[ https://issues.apache.org/jira/browse/GOBBLIN-492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-492. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2363 [https://github.com/apache/incubator-gobblin/pull/2363] > Make LoopingDatasetFinderSource easy to embed different Iterator > > > Key: GOBBLIN-492 > URL: https://issues.apache.org/jira/browse/GOBBLIN-492 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Assignee: Lei Sun >Priority: Major > Fix For: 0.13.0 > > > Refactoring LoopingDatasetFinderSource to make it possible to embed with > iterators other than DeepIterator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (GOBBLIN-499) Log the job name with the tracking URL for easier debugging
Hung Tran created GOBBLIN-499: - Summary: Log the job name with the tracking URL for easier debugging Key: GOBBLIN-499 URL: https://issues.apache.org/jira/browse/GOBBLIN-499 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Assignee: Hung Tran The MR tracking URL is printed in the logs with no association with the job name. This makes it difficult to find the tracking URL for debugging failed jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-499) Log the job name with the tracking URL for easier debugging
[ https://issues.apache.org/jira/browse/GOBBLIN-499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-499. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2369 [https://github.com/apache/incubator-gobblin/pull/2369] > Log the job name with the tracking URL for easier debugging > --- > > Key: GOBBLIN-499 > URL: https://issues.apache.org/jira/browse/GOBBLIN-499 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran >Priority: Major > Fix For: 0.13.0 > > > The MR tracking URL is printed in the logs with no association with the job > name. This makes it difficult to find the tracking URL for debugging failed > jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-510) Decouple JobExecutionLauncher and JobExecutionDriver
[ https://issues.apache.org/jira/browse/GOBBLIN-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-510. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2380 [https://github.com/apache/incubator-gobblin/pull/2380] > Decouple JobExecutionLauncher and JobExecutionDriver > > > Key: GOBBLIN-510 > URL: https://issues.apache.org/jira/browse/GOBBLIN-510 > Project: Apache Gobblin > Issue Type: New Feature >Reporter: Kuai Yu >Assignee: Kuai Yu >Priority: Major > Fix For: 0.13.0 > > > Today JobExecutionLauncher and JobExecutionDriver is coupled. It means when > JobExecutionLauncher invokes launchJob, a JobExecutionDriver is immediately > return. This is not good for gobblin cluster because the Launcher might > running in manager node but the actual driver logic is running on worker > node. We need some refactoring to allow us decouple these two. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-511) Fix Findbugs warnings in Gobblin Service
[ https://issues.apache.org/jira/browse/GOBBLIN-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-511. --- Resolution: Fixed Issue resolved by pull request #2381 [https://github.com/apache/incubator-gobblin/pull/2381] > Fix Findbugs warnings in Gobblin Service > > > Key: GOBBLIN-511 > URL: https://issues.apache.org/jira/browse/GOBBLIN-511 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-service >Affects Versions: 0.13.0 >Reporter: Sudarshan Vasudevan >Assignee: Sudarshan Vasudevan >Priority: Major > Fix For: 0.13.0 > > > Fix findbugs warnings introduced by PR > https://github.com/apache/incubator-gobblin/pull/2361. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (GOBBLIN-513) Add support for queryAll when using the Salesforce bulk API
Hung Tran created GOBBLIN-513: - Summary: Add support for queryAll when using the Salesforce bulk API Key: GOBBLIN-513 URL: https://issues.apache.org/jira/browse/GOBBLIN-513 Project: Apache Gobblin Issue Type: Task Reporter: Hung Tran Assignee: Hung Tran The bulk API operation mode of 'query' does not fetch archived rows. Add the option to enable the 'queryAll' operation mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-489) Implement PusherFactory
[ https://issues.apache.org/jira/browse/GOBBLIN-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-489. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2359 [https://github.com/apache/incubator-gobblin/pull/2359] > Implement PusherFactory > --- > > Key: GOBBLIN-489 > URL: https://issues.apache.org/jira/browse/GOBBLIN-489 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen >Priority: Major > Fix For: 0.13.0 > > > A `PusherFactory` creates a `Pusher`. Changes are: > * `PusherFactory` and gobblin scope specific factory > `GobblinScopePusherFactory` > * Load broker config from configurable multiple namespaces besides > `gobblin.broker` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-412) Compression parameters are not propagated to Hadoop
[ https://issues.apache.org/jira/browse/GOBBLIN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-412. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2386 [https://github.com/apache/incubator-gobblin/pull/2386] > Compression parameters are not propagated to Hadoop > --- > > Key: GOBBLIN-412 > URL: https://issues.apache.org/jira/browse/GOBBLIN-412 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-compaction >Affects Versions: 0.11.0, 0.12.0, 0.13.0 >Reporter: Sushant Pandey >Assignee: Issac Buenrostro >Priority: Minor > Fix For: 0.13.0 > > > Parameters to control compression- > * *mapreduce.output.fileoutputformat.compress* > * *mapreduce.output.fileoutputformat.compress.codec* > * *mapreduce.output.fileoutputformat.compress.type* > are not passed on to Hadoop from compaction job configuration file. In > effect, these parameter's value are always picked up from mapred-site.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-490) Add planning job execution launcher
[ https://issues.apache.org/jira/browse/GOBBLIN-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-490. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2360 [https://github.com/apache/incubator-gobblin/pull/2360] > Add planning job execution launcher > > > Key: GOBBLIN-490 > URL: https://issues.apache.org/jira/browse/GOBBLIN-490 > Project: Apache Gobblin > Issue Type: New Feature >Reporter: Kuai Yu >Assignee: Kuai Yu >Priority: Major > Fix For: 0.13.0 > > > This new job launcher will forward the original job to one of the > GobblinTaskRunner(s). Instead of executing the task driver logic on > GobblinClusterManager, the task driver logic now can be run on > GobblinTaskRunner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-505) Implement a Git-based FlowGraph Monitor
[ https://issues.apache.org/jira/browse/GOBBLIN-505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-505. --- Resolution: Fixed Issue resolved by pull request #2382 [https://github.com/apache/incubator-gobblin/pull/2382] > Implement a Git-based FlowGraph Monitor > --- > > Key: GOBBLIN-505 > URL: https://issues.apache.org/jira/browse/GOBBLIN-505 > Project: Apache Gobblin > Issue Type: New Feature > Components: gobblin-service >Affects Versions: 0.13.0 >Reporter: Sudarshan Vasudevan >Assignee: Sudarshan Vasudevan >Priority: Major > Fix For: 0.13.0 > > > Create a Git-based FlowGraph monitoring service. This service monitors for > changes to a git repo that backs the FlowGraph. The changes include > addition/deletion/modification of either a DataNode or FlowEdge file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (GOBBLIN-513) Add support for queryAll when using the Salesforce bulk API
[ https://issues.apache.org/jira/browse/GOBBLIN-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hung Tran resolved GOBBLIN-513. --- Resolution: Fixed Fix Version/s: 0.13.0 Issue resolved by pull request #2384 [https://github.com/apache/incubator-gobblin/pull/2384] > Add support for queryAll when using the Salesforce bulk API > --- > > Key: GOBBLIN-513 > URL: https://issues.apache.org/jira/browse/GOBBLIN-513 > Project: Apache Gobblin > Issue Type: Task >Reporter: Hung Tran >Assignee: Hung Tran >Priority: Major > Fix For: 0.13.0 > > > The bulk API operation mode of 'query' does not fetch archived rows. Add the > option to enable the 'queryAll' operation mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
