This is an automated email from the ASF dual-hosted git repository.
hutran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-gobblin.git
The following commit(s) were added to refs/heads/master by this push:
new a6c5134 [GOBBLIN-719] fix invalid git links for classes in docs
a6c5134 is described below
commit a6c51342afb40f3a9859ec98814f426b74fcff87
Author: Jay Sen <[email protected]>
AuthorDate: Wed Apr 10 10:29:28 2019 -0700
[GOBBLIN-719] fix invalid git links for classes in docs
[GOBBLIN-719] fix invalid git links for classes in
docs
fixing all file paths and linksA
Closes #2586 from jhsenjaliya/master
---
gobblin-admin/README.md | 2 +-
.../apache/gobblin/source/extractor/Extractor.java | 2 +-
.../cluster/GobblinClusterConfigurationKeys.java | 2 +-
gobblin-docs/Getting-Started.md | 20 +++++++-------
.../adaptors/Hive-Avro-To-ORC-Converter.md | 10 +++----
gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md | 14 +++++-----
gobblin-docs/case-studies/Publishing-Data-to-S3.md | 4 +--
gobblin-docs/case-studies/Writing-ORC-Data.md | 4 +--
.../developer-guide/Documentation-Architecture.md | 2 +-
gobblin-docs/metrics/Existing-Reporters.md | 18 ++++++------
.../metrics/Gobblin-Metrics-Architecture.md | 14 +++++-----
...-generation-instrumentation-for-applications.md | 2 +-
gobblin-docs/metrics/Implementing-New-Reporters.md | 2 +-
gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md | 6 ++--
gobblin-docs/sinks/AvroHdfsDataWriter.md | 2 +-
gobblin-docs/sinks/Couchbase-Writer.md | 6 ++--
gobblin-docs/sinks/Kafka.md | 4 +--
gobblin-docs/sources/QueryBasedSource.md | 16 +++++------
gobblin-docs/sources/RestApiSource.md | 4 +--
gobblin-docs/user-guide/Building-Gobblin.md | 2 +-
gobblin-docs/user-guide/Compaction.md | 2 +-
gobblin-docs/user-guide/Gobblin-Deployment.md | 2 +-
gobblin-docs/user-guide/Gobblin-Schedulers.md | 17 ++++++------
gobblin-docs/user-guide/Gobblin-on-Yarn.md | 32 +++++++++++-----------
gobblin-docs/user-guide/Hive-Registration.md | 18 ++++++------
.../user-guide/Job-Execution-History-Store.md | 2 +-
gobblin-docs/user-guide/Monitoring.md | 2 +-
.../user-guide/State-Management-and-Watermarks.md | 8 +++---
.../Working-with-Job-Configuration-Files.md | 2 +-
.../user-guide/Working-with-the-ForkOperator.md | 20 +++++++-------
gradle/scripts/computeVersions.gradle | 2 +-
mkdocs.yml | 2 +-
32 files changed, 123 insertions(+), 122 deletions(-)
diff --git a/gobblin-admin/README.md b/gobblin-admin/README.md
index 1a9a1f9..0d455b5 100644
--- a/gobblin-admin/README.md
+++ b/gobblin-admin/README.md
@@ -5,7 +5,7 @@ development, will likely change, and have many new features
planned. The
two interfaces provided are a command line interface and a GUI,
accessible via a web server. The current state of the interfaces relies
on the [Job Execution History
-Store](https://github.com/linkedin/gobblin/wiki/Job%20Execution%20History%20Store),
+Store](https://gobblin.readthedocs.io/en/latest/user-guide/Job-Execution-History-Store/),
which must be enabled and running for the interfaces to work.
## CLI
diff --git
a/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java
b/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java
index 2ea900b..9eb31a7 100644
---
a/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java
+++
b/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java
@@ -87,7 +87,7 @@ public interface Extractor<S, D> extends Closeable {
*
* @return the calculated high watermark
* @deprecated there is no longer support for reporting the high watermark
via this method, please see
- * <a
href="https://github.com/linkedin/gobblin/wiki/Watermarks";>Watermarks</a> for
more information.
+ * <a
href="https://gobblin.readthedocs.io/en/latest/user-guide/State-Management-and-Watermarks/";>Watermarks</a>
for more information.
*/
@Deprecated
long getHighWatermark();
diff --git
a/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java
b/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java
index a760c65..f8604fc 100644
---
a/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java
+++
b/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java
@@ -101,7 +101,7 @@ public class GobblinClusterConfigurationKeys {
* A path pointing to a directory that contains job execution files to be
executed by Gobblin. This directory can
* have a nested structure.
*
- * @see <a
href="https://github.com/linkedin/gobblin/wiki/Working-with-Job-Configuration-Files";>Job
Config Files</a>
+ * @see <a
href="https://gobblin.readthedocs.io/en/latest/user-guide/Working-with-Job-Configuration-Files/";>Job
Config Files</a>
*/
public static final String JOB_CONF_PATH_KEY = GOBBLIN_CLUSTER_PREFIX +
"job.conf.path";
public static final String INPUT_WORK_UNIT_DIR_NAME = "_workunits";
diff --git a/gobblin-docs/Getting-Started.md b/gobblin-docs/Getting-Started.md
index f8cfbad..d575997 100644
--- a/gobblin-docs/Getting-Started.md
+++ b/gobblin-docs/Getting-Started.md
@@ -19,7 +19,7 @@ There are two way to obtain a distribution tarball
## Downloading a Pre-Built Distribution
-Download the latest Gobblin release from the [Release
Page](https://github.com/linkedin/gobblin/releases). You will want to download
the `gobblin-distribution-[RELEASE-VERSION].tar.gz` file.
+Download the latest Gobblin release from the [Release
Page](https://github.com/apache/incubator-gobblin/releases). You will want to
download the `gobblin-distribution-[RELEASE-VERSION].tar.gz` file.
## Building a Distribution
@@ -84,17 +84,17 @@ For this example, we will once again run the Wikipedia
example. The records will
## Preliminary
-Each Gobblin job minimally involves several constructs, e.g.
[Source](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/Source.java),
[Extractor](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/extractor/Extractor.java),
[DataWriter](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/writer/DataWriter.java)
and [DataPublisher](https://github.com/linkedin/gobblin/blob/master/gobb [...]
+Each Gobblin job minimally involves several constructs, e.g.
[Source](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/Source.java),
[Extractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java),
[DataWriter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/writer/DataWriter.java)
and [DataPublis [...]
-Some of the classes relevant to this example include
[WikipediaSource](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/java/gobblin/example/wikipedia/WikipediaSource.java),
[WikipediaExtractor](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/java/gobblin/example/wikipedia/WikipediaExtractor.java),
[WikipediaConverter](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/java/gobblin/example/wikipedia/WikipediaConverter
[...]
+Some of the classes relevant to this example include
[WikipediaSource](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaSource.java),
[WikipediaExtractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/java/org/apache/gobblin/example/wikipedia/WikipediaExtractor.java),
[WikipediaConverter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/jav
[...]
-To run Gobblin in standalone daemon mode we need a Gobblin configuration file
(such as uses
[gobblin-standalone.properties](https://github.com/linkedin/gobblin/blob/master/conf/gobblin-standalone.properties)).
And for each job we wish to run, we also need a job configuration file (such
as
[wikipedia.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)).
The Gobblin configuration file, which is passed to Gobblin as a command line
argumen [...]
+To run Gobblin in standalone daemon mode we need a Gobblin configuration file
(such as uses
[gobblin-standalone.properties](https://github.com/apache/incubator-gobblin/blob/master/conf/gobblin-standalone-v2.properties)).
And for each job we wish to run, we also need a job configuration file (such
as
[wikipedia.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)).
The Gobblin configuration file, which is passed to Gobblin as a c [...]
A list of commonly used configuration properties can be found here:
[Configuration Properties
Glossary](user-guide/Configuration-Properties-Glossary).
## Steps
-* Create a folder to store the job configuration file. Put
[wikipedia.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)
in this folder, and set environment variable `GOBBLIN_JOB_CONFIG_DIR` to point
to this folder. Also, make sure that the environment variable `JAVA_HOME` is
set correctly.
+* Create a folder to store the job configuration file. Put
[wikipedia.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)
in this folder, and set environment variable `GOBBLIN_JOB_CONFIG_DIR` to point
to this folder. Also, make sure that the environment variable `JAVA_HOME` is
set correctly.
* Create a folder as Gobblin's working directory. Gobblin will write job
output as well as other information there, such as locks and state-store (for
more information, see the [Standalone
Deployment](user-guide/Gobblin-Deployment#Standalone-Deployment) page). Set
environment variable `GOBBLIN_WORK_DIR` to point to that folder.
@@ -106,7 +106,7 @@ A list of commonly used configuration properties can be
found here: [Configurati
bin/gobblin-standalone.sh start
```
-This script will launch Gobblin and pass the Gobblin configuration file
([gobblin-standalone.properties](https://github.com/linkedin/gobblin/blob/master/conf/gobblin-standalone.properties))
as an argument.
+This script will launch Gobblin and pass the Gobblin configuration file
([gobblin-standalone.properties](https://github.com/apache/incubator-gobblin/blob/master/conf/gobblin-standalone-v2.properties))
as an argument.
The job log, which contains the progress and status of the job, will be
written into `logs/gobblin-current.log` (to change where the log is written,
modify the Log4j configuration file `conf/log4j-standalone.xml`). Stdout will
be written into `nohup.out`.
@@ -157,12 +157,12 @@ java -jar avro-tools-1.8.1.jar tojson --pretty
[job_output].avro > output.json
`output.json` will contain all retrieved records in JSON format.
-Note that since this job configuration file we used
([wikipedia.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull))
doesn't specify a job schedule, the job will run immediately and will run only
once. To schedule a job to run at a certain time and/or repeatedly, set the
`job.schedule` property with a cron-based syntax. For example, `job.schedule=0
0/2 * * * ?` will run the job every two minutes. See [this
link](http://www.quartz-sched [...]
+Note that since this job configuration file we used
([wikipedia.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull))
doesn't specify a job schedule, the job will run immediately and will run only
once. To schedule a job to run at a certain time and/or repeatedly, set the
`job.schedule` property with a cron-based syntax. For example, `job.schedule=0
0/2 * * * ?` will run the job every two minutes. See [this
link](http://www.quar [...]
# Other Example Jobs
-Besides the Wikipedia example, we have another example job
[SimpleJson](https://github.com/linkedin/gobblin/tree/master/gobblin-example/src/main/java/gobblin/example/simplejson),
which extracts records from JSON files and store them in Avro files.
+Besides the Wikipedia example, we have another example job
[SimpleJson](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/simplejson.pull),
which extracts records from JSON files and store them in Avro files.
-To create your own jobs, simply implement the relevant interfaces such as
[Source](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/Source.java),
[Extractor](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/extractor/Extractor.java),
[Converter](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/converter/Converter.java)
and [DataWriter](https://github.com/linkedin/gobblin/blob/ [...]
+To create your own jobs, simply implement the relevant interfaces such as
[Source](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/Source.java),
[Extractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Extractor.java),
[Converter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/converter/Converter.java)
a [...]
-On a side note: while users are free to directly implement the Extractor
interface (e.g., WikipediaExtractor), Gobblin also provides several extractor
implementations based on commonly used protocols, e.g.,
[KafkaExtractor](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaExtractor.java),
[RestApiExtractor](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/resta
[...]
+On a side note: while users are free to directly implement the Extractor
interface (e.g., WikipediaExtractor), Gobblin also provides several extractor
implementations based on commonly used protocols, e.g.,
[KafkaExtractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java),
[RestApiExtractor](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/s
[...]
diff --git a/gobblin-docs/adaptors/Hive-Avro-To-ORC-Converter.md
b/gobblin-docs/adaptors/Hive-Avro-To-ORC-Converter.md
index c03c6e4..98e99bf 100644
--- a/gobblin-docs/adaptors/Hive-Avro-To-ORC-Converter.md
+++ b/gobblin-docs/adaptors/Hive-Avro-To-ORC-Converter.md
@@ -39,16 +39,16 @@ FROM db_name_avro.table_avro WHERE year='2016';
## Source and Extractor
-Gobblin provides
[`HiveSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/source/HiveSource.java)
which is a generic source that connects to the hive metastore and creates
`WorkUnits` for any Hive `Partitions` and `Tables` whitelisted. The
[`HiveConvertExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/extractor/HiveCon
[...]
+Gobblin provides
[`HiveSource`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/source/HiveSource.java)
which is a generic source that connects to the hive metastore and creates
`WorkUnits` for any Hive `Partitions` and `Tables` whitelisted. The
[`HiveConvertExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/manage
[...]
-The `HiveSource` uses the `HiveDatasetFinder` to find all hive tables and
partitions that satisfy a whitelist. For each table/partition it creates a
workunit is the `updateTime` is greater than the `lowWatermark`. By default a
[`PartitionLevelWatermarker`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/watermarker/PartitionLevelWatermarker.java)
is used. This watermarker tracks watermarks for every partition o [...]
+The `HiveSource` uses the `HiveDatasetFinder` to find all hive tables and
partitions that satisfy a whitelist. For each table/partition it creates a
workunit is the `updateTime` is greater than the `lowWatermark`. By default a
[`PartitionLevelWatermarker`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/watermarker/PartitionLevelWatermarker.java)
is used. This watermarker tracks watermarks fo [...]
The `HiveConvertExtractor` builds `QueryBasedHiveConversionEntity`s. The
extractor makes necessary calls to the Hive Metastore to get table/partition
metadata. The metadata is then wrapped into a `QueryBasedHiveConversionEntity`.
## Converter
The converter builds the Hive DDLs/DMLs required to perform the Avro to ORC
conversion. Gobblin supports conversion of Avro to both flattened ORC and
nested ORC.
-The abstract converter
[`AbstractAvroToOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java)
builds DDLs/DMLs for any destination ORC format. Concrete subclass
[`HiveAvroToFlattenedOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/HiveAvroToFlattenedOrcConverte
[...]
+The abstract converter
[`AbstractAvroToOrcConverter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java)
builds DDLs/DMLs for any destination ORC format. Concrete subclass
[`HiveAvroToFlattenedOrcConverter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/co
[...]
The converter builds the following different DDLs/DMLs
@@ -63,12 +63,12 @@ The converter builds the following different DDLs/DMLs
## Writer
-The writer in this context executes the Hive DDLs/DMLs generated by the
converter.
[`HiveQueryExecutionWriter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/writer/HiveQueryExecutionWriter.java)
uses Hive JDBC connector to execute the DDLs. The DDLs write ORC data into
staging tables. After the writer has completed
`HiveQueryExecutionWriter#write()`, ORC data will be available in the staging
tables.
+The writer in this context executes the Hive DDLs/DMLs generated by the
converter.
[`HiveQueryExecutionWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/writer/HiveQueryExecutionWriter.java)
uses Hive JDBC connector to execute the DDLs. The DDLs write ORC data into
staging tables. After the writer has completed
`HiveQueryExecutionWriter#write()`, ORC data will be available in the stagi
[...]
## Publisher
-The publisher
[`HiveConvertPublisher`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/publisher/HiveConvertPublisher.java)
executes hive DDLs to publish staging ORC tables to final ORC tables. The
publisher also cleans up staging tables.
+The publisher
[`HiveConvertPublisher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/publisher/HiveConvertPublisher.java)
executes hive DDLs to publish staging ORC tables to final ORC tables. The
publisher also cleans up staging tables.
By default publishing happens per dataset (dataset = table in this context).
If a dataset fails, other datasets will still be published but the job will
fail. The commit policy is configurable.
diff --git a/gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md
b/gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md
index feaca55..352b7e3 100644
--- a/gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md
+++ b/gobblin-docs/case-studies/Kafka-HDFS-Ingestion.md
@@ -107,17 +107,17 @@ After the job finishes, the job output file will be in
`/gobblintest/job-output/
## Source and Extractor
-Gobblin provides two abstract classes,
[`KafkaSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaSource.java)
and
[`KafkaExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaExtractor.java).
`KafkaSource` creates a workunit for each Kafka topic partition to be pulled,
then merges and groups the workunits based on the desired number of [...]
+Gobblin provides two abstract classes,
[`KafkaSource`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSource.java)
and
[`KafkaExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaExtractor.java).
`KafkaSource` creates a workunit for each Kafka topic partit [...]
To use them in a Kafka-HDFS ingestion job, one should subclass
`KafkaExtractor` and implement method `decodeRecord(MessageAndOffset)`, which
takes a `MessageAndOffset` object pulled from the Kafka broker and decodes it
into a desired object. One should also subclass `KafkaSource` and implement
`getExtractor(WorkUnitState)` which should return an instance of the Extractor
class.
-As examples, take a look at
[`KafkaSimpleSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaSimpleSource.java),
[`KafkaSimpleExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/kafka/KafkaSimpleExtractor.java),
and
[`KafkaAvroExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/k
[...]
+As examples, take a look at
[`KafkaSimpleSource`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSimpleSource.java),
[`KafkaSimpleExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaSimpleExtractor.java),
and [`KafkaAvroExtractor`](https://github.com/apac [...]
`KafkaSimpleExtractor` simply returns the payload of the `MessageAndOffset`
object as a byte array. A job that uses `KafkaSimpleExtractor` may use a
`Converter` to convert the byte array to whatever format desired. For example,
if the desired output format is JSON, one may implement an
`ByteArrayToJsonConverter` to convert the byte array to JSON. Alternatively one
may implement a `KafkaJsonExtractor`, which extends `KafkaExtractor` and
convert the `MessageAndOffset` object into a JSON ob [...]
## Writer and Publisher
-Any desired writer and publisher can be used, e.g., one may use the
[`AvroHdfsDataWriter`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/writer/AvroHdfsDataWriter.java)
and the
[`BaseDataPublisher`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/publisher/BaseDataPublisher.java),
similar as the [Wikipedia example job](../Getting-Started). If plain text
output file is desired, one may use [`SimpleDataWriter`](https:/ [...]
+Any desired writer and publisher can be used, e.g., one may use the
[`AvroHdfsDataWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/AvroHdfsDataWriter.java)
and the
[`BaseDataPublisher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisher.java),
similar as the [Wikipedia example
job](https://github.com/apache/incubator-gobblin/blob/master/gobbl [...]
# Job Config Properties
@@ -145,7 +145,7 @@ extract.limit.timeLimitTimeunit=minutes
## Task Level Metrics
-Task level metrics can be created in `Extractor`, `Converter` and `Writer` by
extending
[`InstrumentedExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/instrumented/extractor/InstrumentedExtractor.java),
[`InstrumentedConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/instrumented/converter/InstrumentedConverter.java)
and
[`InstrumentedDataWriter`](https://github.com/linkedin/gobblin/blob/master/gobb
[...]
+Task level metrics can be created in `Extractor`, `Converter` and `Writer` by
extending
[`InstrumentedExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/extractor/InstrumentedExtractor.java),
[`InstrumentedConverter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/converter/InstrumentedConverter.java)
and [`InstrumentedDataWriter`](http [...]
For example, `KafkaExtractor` extends `InstrumentedExtractor`. So you can do
the following in subclasses of `KafkaExtractor`:
@@ -158,11 +158,11 @@ Besides Counter, Meter and Histogram are also supported.
## Task Level Events
-Task level events can be submitted by creating an
[`EventSubmitter`](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/event/EventSubmitter.java)
instance and using `EventSubmitter.submit()` or
`EventSubmitter.getTimingEvent()`.
+Task level events can be submitted by creating an
[`EventSubmitter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/event/EventSubmitter.java)
instance and using `EventSubmitter.submit()` or
`EventSubmitter.getTimingEvent()`.
## Job Level Metrics
-To create job level metrics, one may extend
[`AbstractJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/AbstractJobLauncher.java)
and create metrics there. For example:
+To create job level metrics, one may extend
[`AbstractJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java)
and create metrics there. For example:
```
Optional<JobMetrics> jobMetrics = this.jobContext.getJobMetricsOptional();
@@ -186,7 +186,7 @@ For more details about metrics, events and reporting them,
please see Gobblin Me
For each topic partition that should be ingested, `KafkaSource` first
retrieves the last offset pulled by the previous run, which should be the first
offset of the current run. It also retrieves the earliest and latest offsets
currently available from the Kafka cluster and verifies that the first offset
is between the earliest and the latest offsets. The latest offset is the last
offset to be pulled by the current workunit. Since new records may be
constantly published to Kafka and old r [...]
-For each partition, after the first and last offsets are determined, a
workunit is created. If the number of Kafka partitions exceeds the desired
number of workunits specified by property `mr.job.max.mappers`, `KafkaSource`
will merge and group them into `n`
[`MultiWorkUnit`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/workunit/MultiWorkUnit.java)s
where `n=mr.job.max.mappers`. This is done using
[`KafkaWorkUnitPacker`](https://github.com/link [...]
+For each partition, after the first and last offsets are determined, a
workunit is created. If the number of Kafka partitions exceeds the desired
number of workunits specified by property `mr.job.max.mappers`, `KafkaSource`
will merge and group them into `n`
[`MultiWorkUnit`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/workunit/MultiWorkUnit.java)s
where `n=mr.job.max.mappers`. This is done using [`KafkaWorkUnitPacker`](http
[...]
## Single-Level Packing
diff --git a/gobblin-docs/case-studies/Publishing-Data-to-S3.md
b/gobblin-docs/case-studies/Publishing-Data-to-S3.md
index 8d1e89a..9866e85 100644
--- a/gobblin-docs/case-studies/Publishing-Data-to-S3.md
+++ b/gobblin-docs/case-studies/Publishing-Data-to-S3.md
@@ -95,7 +95,7 @@ Go to the [S3 dashboard](https://console.aws.amazon.com/s3)
## Setting Up Gobblin on EC2
* Download and Build Gobblin Locally
- * On your local machine, clone the [Gobblin
repository](https://github.com/linkedin/gobblin): `git clone
[email protected]:linkedin/gobblin.git` (this assumes you have
[Git](https://en.wikipedia.org/wiki/Git_(software)) installed locally)
+ * On your local machine, clone the [Gobblin
repository](https://github.com/apache/incubator-gobblin): `git clone
[email protected]:linkedin/gobblin.git` (this assumes you have
[Git](https://en.wikipedia.org/wiki/Git_(software)) installed locally)
* Build Gobblin using the following commands (it is important to use
Hadoop version 2.6.0 as it includes the `s3a` file system implementation):
```
cd gobblin
@@ -132,7 +132,7 @@ Assuming we are running Gobblin in [standalone
mode](../user-guide/Gobblin-Deplo
Assuming we want Gobblin to run in standalone mode, follow the usual steps for
[standalone deployment](../user-guide/Gobblin-Deployment#Standalone-Deployment).
-For the sake of this walkthrough, we will launch the Gobblin [wikipedia
example](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull).
Directions on how to run this example can be found [here](../Getting-Started).
The command to launch Gobblin should look similar to:
+For the sake of this walkthrough, we will launch the Gobblin [wikipedia
example](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull).
Directions on how to run this example can be found [here](../Getting-Started).
The command to launch Gobblin should look similar to:
```
sh bin/gobblin-standalone.sh start --workdir /home/ec2-user/gobblin-dist/work
--logdir /home/ec2-user/gobblin-dist/logs --conf
/home/ec2-user/gobblin-dist/config
```
diff --git a/gobblin-docs/case-studies/Writing-ORC-Data.md
b/gobblin-docs/case-studies/Writing-ORC-Data.md
index 26da218..411409a 100644
--- a/gobblin-docs/case-studies/Writing-ORC-Data.md
+++ b/gobblin-docs/case-studies/Writing-ORC-Data.md
@@ -4,7 +4,7 @@
# Introduction
-Gobblin is capable of writing data to ORC files by leveraging Hive's SerDe
library. Gobblin has native integration with Hive SerDe's library via the
[HiveSerDeWrapper](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/HiveSerDeManager.java)
class.
+Gobblin is capable of writing data to ORC files by leveraging Hive's SerDe
library. Gobblin has native integration with Hive SerDe's library via the
[HiveSerDeWrapper](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveSerDeManager.java)
class.
This document will briefly explain how Gobblin integrates with Hive's SerDe
library, and show an example of writing ORC files.
@@ -20,7 +20,7 @@ Gobblin integrates with the Hive SerDe's in a few different
places. Here is a li
# Writing to an ORC File
-An end-to-end example of writing to an ORC file is provided in the
configuration found
[here](https://github.com/linkedin/gobblin/tree/master/gobblin-example/src/main/resources/wikipedia-orc.pull).
This `.pull` file is almost identical to the Wikipedia example discussed in
the [Getting Started Guide](../Getting-Started.md). The only difference is that
the output is written in ORC instead of Avro. The configuration file mentioned
above can be directly used as a template for writing data t [...]
+An end-to-end example of writing to an ORC file is provided in the
configuration found
[here](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-orc.pull).
This `.pull` file is almost identical to the Wikipedia example discussed in
the [Getting Started Guide](../Getting-Started.md). The only difference is that
the output is written in ORC instead of Avro. The configuration file mentioned
above can be directly used as a template for writin [...]
* `converter.classes` requires two additional converters:
`gobblin.converter.avro.AvroRecordToAvroWritableConverter` and
`gobblin.converter.serde.HiveSerDeConverter`
* The output of the first converter (the `WikipediaConverter`) returns
Avro `GenericRecord`s
diff --git a/gobblin-docs/developer-guide/Documentation-Architecture.md
b/gobblin-docs/developer-guide/Documentation-Architecture.md
index 80de729..15a04bc 100644
--- a/gobblin-docs/developer-guide/Documentation-Architecture.md
+++ b/gobblin-docs/developer-guide/Documentation-Architecture.md
@@ -21,4 +21,4 @@ ReadTheDocs is an open source, free tool that can build
documentation in a GitHu
# Additional Information
-For more information on how this architecture was decided and the different
tradeoffs between other documentation services, check out the original PR:
https://github.com/linkedin/gobblin/pull/788 and GitHub Issue:
https://github.com/linkedin/gobblin/issues/724
+For more information on how this architecture was decided and the different
tradeoffs between other documentation services, check out the original PR:
https://github.com/apache/incubator-gobblin/pull/788
diff --git a/gobblin-docs/metrics/Existing-Reporters.md
b/gobblin-docs/metrics/Existing-Reporters.md
index e573941..909f285 100644
--- a/gobblin-docs/metrics/Existing-Reporters.md
+++ b/gobblin-docs/metrics/Existing-Reporters.md
@@ -6,15 +6,15 @@ Table of Contents
Metric Reporters
================
-* [Output Stream
Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/OutputStreamReporter.java):
allows printing metrics to any OutputStream, including STDOUT and files.
-* [Kafka
Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/kafka/KafkaReporter.java):
emits metrics to Kafka topic as Json messages.
-* [Kafka Avro
Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/kafka/KafkaAvroReporter.java):
emits metrics to Kafka topic as Avro messages with schema
[MetricReport](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/avro/MetricReport.avsc).
-* [Graphite
Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/graphite/GraphiteReporter.java):
emits metrics to Graphite. This reporter has a different, deprecated
construction API included in its javadoc.
-* [Influx DB
Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/influxdb/InfluxDBReporter.java):
emits metrics to Influx DB. This reporter has a different, deprecated
construction API included in its javadoc.
-* [Hadoop Counter
Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/hadoop/HadoopCounterReporter.java):
emits metrics as Hadoop counters at the end of the execution. Available for
old and new Hadoop API. This reporter has a different, deprecated construction
API included in its javadoc. Due to limits on the number of Hadoop counters
that can be created, this reporter is not recommended except for applications
with very few metrics.
+* [Output Stream
Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/OutputStreamReporter.java):
allows printing metrics to any OutputStream, including STDOUT and files.
+* [Kafka
Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaReporter.java):
emits metrics to Kafka topic as Json messages.
+* [Kafka Avro
Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaAvroReporter.java):
emits metrics to Kafka topic as Avro messages with schema
[MetricReport](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/MetricReport.avsc).
+* [Graphite
Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-graphite/src/main/java/org/apache/gobblin/metrics/graphite/GraphiteReporter.java):
emits metrics to Graphite. This reporter has a different, deprecated
construction API included in its javadoc.
+* [Influx DB
Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-influxdb/src/main/java/org/apache/gobblin/metrics/influxdb/InfluxDBReporter.java):
emits metrics to Influx DB. This reporter has a different, deprecated
construction API included in its javadoc.
+* [Hadoop Counter
Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-metrics-hadoop/src/main/java/org/apache/gobblin/metrics/hadoop/HadoopCounterReporter.java):
emits metrics as Hadoop counters at the end of the execution. Available for
old and new Hadoop API. This reporter has a different, deprecated construction
API included in its javadoc. Due to limits on the number of Hadoop counters
that can be created, this reporter is not recommended except [...]
Event Reporters
===============
-* [Output Stream Event
Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/OutputStreamEventReporter.java):
Emits events to any output stream, including STDOUT and files.
-* [Kafka Event
Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/kafka/KafkaEventReporter.java):
Emits events to Kafka topic as Json messages.
-* [Kafka Avro Event
Reporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/kafka/KafkaAvroEventReporter.java):
Emits events to Kafka topic as Avro messages using the schema
[GobblinTrackingEvent](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/avro/GobblinTrackingEvent.avsc).
+* [Output Stream Event
Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/OutputStreamEventReporter.java):
Emits events to any output stream, including STDOUT and files.
+* [Kafka Event
Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaEventReporter.java):
Emits events to Kafka topic as Json messages.
+* [Kafka Avro Event
Reporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/metrics/kafka/KafkaAvroEventReporter.java):
Emits events to Kafka topic as Avro messages using the schema
[GobblinTrackingEvent](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinTrackingEvent.avsc).
diff --git a/gobblin-docs/metrics/Gobblin-Metrics-Architecture.md
b/gobblin-docs/metrics/Gobblin-Metrics-Architecture.md
index 4e15523..f479ac4 100644
--- a/gobblin-docs/metrics/Gobblin-Metrics-Architecture.md
+++ b/gobblin-docs/metrics/Gobblin-Metrics-Architecture.md
@@ -19,15 +19,15 @@ Each metric context contains the following instance
variables:
* A `String` `name`. The name is not used by the core metrics engine, but can
be accessed by users to identify the context.
* A reference to the parent metric context, or null if it has no parent.
* A list of children metric context references, stored as soft references.
-* An object of type
[Tagged](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/Tagged.java)
containing the tags for this metric context.
-* A `Set` of notification targets. Notification targets are objects of type
[Function](http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/base/Function.html)<[Notification](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/notification/Notification.java),
Void> which are all called every time there is a new notification.
Notifications can be submitted to the Metric Context using the method
`sendNotification(Notification)`. [...]
+* An object of type
[Tagged](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/Tagged.java)
containing the tags for this metric context.
+* A `Set` of notification targets. Notification targets are objects of type
[Function](https://google.github.io/guava/releases/15.0/api/docs/com/google/common/base/Function.html)<[Notification](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/notification/Notification.java),
Void> which are all called every time there is a new notification.
Notifications can be submitted to the Metric Context using [...]
* A lazily instantiated `ExecutorService` used for asynchronously executing
the notification targets. The executor service will only be started the first
time there is a notification and the number of notification targets is positive.
* A `ConcurrentMap` from metric names to `Metric` for all metrics registered
in this Metric Context. Metrics can be added to this map using the
`register(Metric)`, `register(String, Metric)`, or `registerAll(MetricSet)`,
although it is recommended to instead use the methods to create and register
the metrics. Metric Context implements getter methods for all metrics, as well
as for each type of metric individually (`getMetrics`, `getGauges`,
`getCounters`, `getHistograms`, `getMeters`, `g [...]
Metrics
=======
-All metrics extend the interface
[ContextAwareMetric](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/ContextAwareMetric.java).
Each metric type in Dropwizard Metrics is extended to a Context Aware type:
`ContextAwareCounter`, `ContextAwareGauge`, `ContextAwareHistogram`,
`ContextAwareMeter`, `ContextAwareTimer`.
+All metrics extend the interface
[ContextAwareMetric](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/ContextAwareMetric.java).
Each metric type in Dropwizard Metrics is extended to a Context Aware type:
`ContextAwareCounter`, `ContextAwareGauge`, `ContextAwareHistogram`,
`ContextAwareMeter`, `ContextAwareTimer`.
Context Aware metrics all always created from the Metric Context where they
will be registered. For example, to get a counter under Metric Context
`context`, the user would call `context.counter("counter.name")`. This method
first checks all registered metrics in the Metric Context to find a counter
with that name, if it succeeds, it simply returns that counter. If a counter
with that name has not been registered in `context`, then a new
`ContextAwareCounter` is created and registered in [...]
@@ -38,16 +38,16 @@ Users can also register objects of type
`com.codahale.metrics.Metric` with any M
Events
======
-Events are objects of type
[GobblinTrackingEvent](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/avro/GobblinTrackingEvent.avsc),
which is a type generated from an Avro schema. Events have:
+Events are objects of type
[GobblinTrackingEvent](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/avro/GobblinTrackingEvent.avsc),
which is a type generated from an Avro schema. Events have:
* A `namespace`.
* A `name`.
* A `timestamp`.
* A `Map<String,String>` of `metadata`.
-Events are submitted using the
`MetricContext#submitEvent(GobblinTrackingEvent)` method. When called, this
method packages the event into an
[EventNotification](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/notification/EventNotification.java)
and submits it to the metric context using the method
`MetricContext#sendNotification(Notification)`. This notification is passed to
all metrics context ancestors. Each notification target of each anc [...]
+Events are submitted using the
`MetricContext#submitEvent(GobblinTrackingEvent)` method. When called, this
method packages the event into an
[EventNotification](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/notification/EventNotification.java)
and submits it to the metric context using the method
`MetricContext#sendNotification(Notification)`. This notification is passed to
all metrics context an [...]
-Events can be created manually using Avro constructors, and using the method
`context.submitEvent(GobblinTrackinEvent)`, but this is unfriendly when trying
to build events incrementally, especially when using metadata. To address this,
users can instead use
[EventSubmitter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/event/EventSubmitter.java)
which is an abstraction around the Avro constructor for GobblinTrackingEvent.
+Events can be created manually using Avro constructors, and using the method
`context.submitEvent(GobblinTrackinEvent)`, but this is unfriendly when trying
to build events incrementally, especially when using metadata. To address this,
users can instead use
[EventSubmitter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/event/EventSubmitter.java)
which is an abstraction around the Avro constructor [...]
Event Submitter
---------------
@@ -59,7 +59,7 @@ Reporters
Reporters export the metrics and/or events of a metric context to a sink.
Reporters extend the interface `com.codahale.metrics.Reporter`. Most reporters
will attach themselves to a Metric Context. The reporter can then navigate the
Metric Context tree where the Metric Context belongs, get tags and metrics, get
notified of events, and export them to the sink.
-The two best entry points for developing reporters are
[RecursiveScheduledMetricReporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java)
and
[EventReporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/EventReporter.java).
These classes do most of the heavy lifting for reporting metrics and events
respectively. They are both scheduled re [...]
+The two best entry points for developing reporters are
[RecursiveScheduledMetricReporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java)
and
[EventReporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java).
These classes do most of t [...]
RecursiveScheduleMetricReporter
-------------------------------
diff --git
a/gobblin-docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md
b/gobblin-docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md
index d26eb2b..9289de2 100644
---
a/gobblin-docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md
+++
b/gobblin-docs/metrics/Gobblin-Metrics-next-generation-instrumentation-for-applications.md
@@ -34,5 +34,5 @@ Each metric context manages a set of metrics (like counters,
timers, meters, and
</p>
<p>
- To learn more about Gobblin Metrics, check out the <a
href="https://github.com/linkedin/gobblin/wiki/Gobblin%20Metrics";>Wiki</a> and
the <a href="https://github.com/linkedin/gobblin";>Gobblin project</a> in Github.
+ To learn more about Gobblin Metrics, check out the <a
href="https://gobblin.readthedocs.io/en/latest/metrics/Gobblin-Metrics/";>Wiki</a>
and the <a href="https://github.com/apache/incubator-gobblin";>Gobblin
project</a> in Github.
</p>
diff --git a/gobblin-docs/metrics/Implementing-New-Reporters.md
b/gobblin-docs/metrics/Implementing-New-Reporters.md
index 42be354..1c79ea5 100644
--- a/gobblin-docs/metrics/Implementing-New-Reporters.md
+++ b/gobblin-docs/metrics/Implementing-New-Reporters.md
@@ -3,7 +3,7 @@ Table of Contents
[TOC]
-The two best entry points for implementing custom reporters are
[RecursiveScheduledMetricReporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java)
and
[EventReporter](https://github.com/linkedin/gobblin/blob/master/gobblin-metrics/src/main/java/gobblin/metrics/reporter/EventReporter.java).
Each of these classes automatically schedules reporting, extracts the correct
metrics, and calls a single m [...]
+The two best entry points for implementing custom reporters are
[RecursiveScheduledMetricReporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/RecursiveScheduledMetricReporter.java)
and
[EventReporter](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/reporter/EventReporter.java).
Each of these cla [...]
In the interest of giving more control to the users, metric and event
reporters are kept separate, allowing users to more easily specify separate
sinks for events and metrics. However, it is possible to implement a single
report that handles both events and metrics.
diff --git a/gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md
b/gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md
index c0321be..3ece6c7 100644
--- a/gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md
+++ b/gobblin-docs/metrics/Metrics-for-Gobblin-ETL.md
@@ -126,13 +126,13 @@ Customizing Instrumentation
Custom constructs
-----------------
-When using a custom construct (for example a custom extractor for your data
source), you will get the above mentioned instrumentation for free. However,
you may want to implement additional metrics. To aid with this, instead of
extending the usual class Extractor, you can extend the class
`gobblin.instrumented.extractor.InstrumentedExtractor`. Similarly, for each
construct there is an instrumented version that allows extension of the default
metrics ([InstrumentedExtractor](https://githu [...]
+When using a custom construct (for example a custom extractor for your data
source), you will get the above mentioned instrumentation for free. However,
you may want to implement additional metrics. To aid with this, instead of
extending the usual class Extractor, you can extend the class
`gobblin.instrumented.extractor.InstrumentedExtractor`. Similarly, for each
construct there is an instrumented version that allows extension of the default
metrics ([InstrumentedExtractor](https://githu [...]
All of the instrumented constructs have Javadoc providing with additional
information. In general, when extending an instrumented construct, you will
have to implement a different method. For example, when extending an
InstrumentedExtractor, instead of implementing `readRecord`, you will implement
`readRecordImpl`. To make this clearer for the user, implementing `readRecord`
will throw a compilation error, and the javadoc of each method specifies the
method that should be implemented.
### Instrumentable Interface
-Instrumented constructs extend the interface
[Instrumentable](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/instrumented/Instrumentable.java).
It contains the following methods:
+Instrumented constructs extend the interface
[Instrumentable](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/instrumented/Instrumentable.java).
It contains the following methods:
* `getMetricContext()`: get the default metric context generated for that
instance of the construct, with all the appropriate tags. Use this metric
context to create any additional metrics.
* `isInstrumentationEnabled()`: returns true if instrumentation is enabled.
@@ -149,6 +149,6 @@ Instrumented constructs have a set of callback methods that
are called at differ
Custom Reporters
----------------
-Besides the reporters implemented by default (file, Kafka, and JMX), users can
add custom reporters to the classpath and instruct Gobblin to use these
reporters. To do this, users should extend the interface
[CustomReporterFactory](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/metrics/CustomReporterFactory.java),
and specify a comma-separated list of CustomReporterFactory classes in the
configuration key `metrics.reporting.custom.builders`.
+Besides the reporters implemented by default (file, Kafka, and JMX), users can
add custom reporters to the classpath and instruct Gobblin to use these
reporters. To do this, users should extend the interface
[CustomReporterFactory](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metrics-libs/gobblin-metrics-base/src/main/java/org/apache/gobblin/metrics/CustomReporterFactory.java),
and specify a comma-separated list of CustomReporterFactory classes in the
configuration key [...]
Gobblin will automatically search for these CustomReporterFactory
implementations, instantiate each one with a parameter-less constructor, and
then call the method `newScheduledReporter(MetricContext, Properties)`, where
the properties contain all of the input configurations supplied to Gobblin.
Gobblin will then manage this `ScheduledReporter`.
diff --git a/gobblin-docs/sinks/AvroHdfsDataWriter.md
b/gobblin-docs/sinks/AvroHdfsDataWriter.md
index f74d6a4..8ac89fb 100644
--- a/gobblin-docs/sinks/AvroHdfsDataWriter.md
+++ b/gobblin-docs/sinks/AvroHdfsDataWriter.md
@@ -9,7 +9,7 @@ Writes Avro records to Avro data files on Hadoop file systems.
writer.builder.class=org.apache.gobblin.writer.AvroDataWriterBuilder
writer.destination.type=HDFS
-For more info, see
[`AvroHdfsDataWriter`](https://github.com/linkedin/gobblin/search?utf8=%E2%9C%93&q=AvroHdfsDataWriter)
+For more info, see
[`AvroHdfsDataWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/AvroHdfsDataWriter.java)
# Configuration
diff --git a/gobblin-docs/sinks/Couchbase-Writer.md
b/gobblin-docs/sinks/Couchbase-Writer.md
index 8b96b30..6527008 100644
--- a/gobblin-docs/sinks/Couchbase-Writer.md
+++ b/gobblin-docs/sinks/Couchbase-Writer.md
@@ -1,4 +1,4 @@
-TODO [Issue #1544](https://github.com/linkedin/gobblin/issues/1544)
+TODO [GOBBLIN-59](https://issues.apache.org/jira/browse/GOBBLIN-59)
# Description
@@ -8,7 +8,7 @@ Writes records to Couchbase.
# Usage
-This writer is part of the gobblin-couchbase [Gobblin
Module](developer-guide/GobblinModules). See
-
[`CouchbaseWriter`](https://github.com/linkedin/gobblin/search?utf8=%E2%9C%93&q=CouchbaseWriter)
for more info.
+This writer is part of the gobblin-couchbase [Gobblin
Module](https://github.com/apache/incubator-gobblin/tree/master/gobblin-modules/gobblin-couchbase).
See
+
[`CouchbaseWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-couchbase/src/main/java/org/apache/gobblin/couchbase/writer/CouchbaseWriter.java)
for more info.
# Configuration
\ No newline at end of file
diff --git a/gobblin-docs/sinks/Kafka.md b/gobblin-docs/sinks/Kafka.md
index af47f4b..10c243f 100644
--- a/gobblin-docs/sinks/Kafka.md
+++ b/gobblin-docs/sinks/Kafka.md
@@ -15,7 +15,7 @@ The Kafka writer allows users to create pipelines that ingest
data from Gobblin
#Steps
-* Edit the
[wikipedia-kafka.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-kafka.pull)
example to get started with setting up ingestion into Kafka. This is a very
similar pipeline to the
[wikipedia.pull](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)
example which pulls pages from 5 titles from Wikipedia to HDFS. The main
differences to note are:
+* Edit the
[wikipedia-kafka.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-kafka.pull)
example to get started with setting up ingestion into Kafka. This is a very
similar pipeline to the
[wikipedia.pull](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)
example which pulls pages from 5 titles from Wikipedia to HDFS. The main
differences to note are:
* The `writer.builder.class` is set to
`gobblin.kafka.writer.KafkaDataWriterBuilder`. This is the class that creates a
Kafka writer.
* The `writer.kafka.topic` is set to `WikipediaExample`. This is the topic
that the writer will write the records to.
* The `writer.kafka.producerConfig.bootstrap.servers` is set to
`localhost:9092`. This is the address of the kafka broker(s) that the writer
must write to.
@@ -53,7 +53,7 @@ INFO [TaskExecutor-0] gobblin.runtime.Fork 261 - Committing
data for fork 0 of
INFO [TaskExecutor-0] gobblin.kafka.writer.KafkaDataWriter 211 -
Successfully committed 20 records.
```
-* To verify that the records have indeed been ingested into Kafka, you can run
a kafka console consumer or run Gobblin's [kafka-console pull
file](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia-kafka.pull)
which prints the events from Kafka onto the console.
+* To verify that the records have indeed been ingested into Kafka, you can run
a kafka console consumer or run Gobblin's [kafka-console pull
file](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/kafka-console.pull)
which prints the events from Kafka onto the console.
#Configuration Details
diff --git a/gobblin-docs/sources/QueryBasedSource.md
b/gobblin-docs/sources/QueryBasedSource.md
index 217f389..baf6b83 100644
--- a/gobblin-docs/sources/QueryBasedSource.md
+++ b/gobblin-docs/sources/QueryBasedSource.md
@@ -1,11 +1,11 @@
[TOC]
# Introduction
-[`QueryBasedSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/QueryBasedSource.java)
+[`QueryBasedSource`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java)
represents a category of sources whose data is pulled by sending queries. A
dataset of a source is identified as a
-[`SourceEntity`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/QueryBasedSource.java#L96).
+[`SourceEntity`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java#L96).
Query can be done by sending HTTP requests or SQL commands. A source often,
but not always, has a corresponding
-[`QueryBasedExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/QueryBasedExtractor.java),
+[`QueryBasedExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedExtractor.java),
which defines the way and implements common routines to extract data from the
source.
# Constructs
@@ -25,7 +25,7 @@ Like other categories of sources, a `QueryBasedSource`
focuses on creating work
- pick up previously failed work units.
At last, it will group several work units as
-[`MultiWorkUnit`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/workunit/MultiWorkUnit.java)
+[`MultiWorkUnit`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/workunit/MultiWorkUnit.java)
according to the `mr.job.max.mappers` configuration (Note: other categories of
source might have a different approach to group work units into
`MultiWorkUnit`).
## `QueryBasedExtractor`
@@ -37,13 +37,13 @@ according to the `mr.job.max.mappers` configuration (Note:
other categories of s
</p>
Currently in Gobblin, depending on how an extractor communicates with a source
-(or [different communication
protocols](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/ProtocolSpecificLayer.java)),
+(or [different communication
protocols](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/ProtocolSpecificLayer.java)),
a `QueryBasedExtractor` falls into 2 categories:
-[`RestApiExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/restapi/RestApiExtractor.java)
+[`RestApiExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiExtractor.java)
and
-[`JdbcExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/jdbc/JdbcExtractor.java).
+[`JdbcExtractor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-sql/src/main/java/org/apache/gobblin/source/jdbc/JdbcExtractor.java).
A specific extractor has to provide some
-[source specific
logic](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/SourceSpecificLayer.java)
+[source specific
logic](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/SourceSpecificLayer.java)
in order to successfully extract information from the source.
### `build`
diff --git a/gobblin-docs/sources/RestApiSource.md
b/gobblin-docs/sources/RestApiSource.md
index 1f25096..ab9dde9 100644
--- a/gobblin-docs/sources/RestApiSource.md
+++ b/gobblin-docs/sources/RestApiSource.md
@@ -3,7 +3,7 @@
# Introduction
A RestApiSource is a [QueryBasedSource](../sources/QueryBasedSource.md) which
uses [RESTful](https://en.wikipedia.org/wiki/Representational_state_transfer)
Api for query. `RestApiExtractor` is a `QueryBasedExtractor` that uses REST to
communicate with the source. To establish the communication,
-a
[`RestApiConnector`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/restapi/RestApiConnector.java)
is
+a
[`RestApiConnector`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/restapi/RestApiConnector.java)
is
required.
# Constructs
@@ -36,7 +36,7 @@ Depends on the routines, [getX], [constructGetXQuery],
[extractXFromResponse] ar
There are other interactions between the `RestApiExtractor` layer and
`SourceSpecificLayer`. The key points are:
-- A
[`ProtocolSpecificLayer`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/extract/ProtocolSpecificLayer.java),
such as
+- A
[`ProtocolSpecificLayer`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/ProtocolSpecificLayer.java),
such as
`RestApiExtractor`, understands the protocol and sets up a routine to
communicate with the source
- A `SourceSpecificLayer`, such as `SalesforceExtractor`, knows the source and
fits into the routine by providing and analyzing source specific information
diff --git a/gobblin-docs/user-guide/Building-Gobblin.md
b/gobblin-docs/user-guide/Building-Gobblin.md
index 1e51cc4..f29e85c 100644
--- a/gobblin-docs/user-guide/Building-Gobblin.md
+++ b/gobblin-docs/user-guide/Building-Gobblin.md
@@ -7,7 +7,7 @@
This page outlines all the options that can be specified when building Gobblin
using Gradle. The typical way of building Gobblin is to first checkout the
code-base from GitHub and then build the code-base using Gradle.
```
-git clone https://github.com/linkedin/gobblin.git
+git clone https://github.com/apache/incubator-gobblin.git
cd gobblin
./gradlew assemble
```
diff --git a/gobblin-docs/user-guide/Compaction.md
b/gobblin-docs/user-guide/Compaction.md
index 5784d39..74bf985 100644
--- a/gobblin-docs/user-guide/Compaction.md
+++ b/gobblin-docs/user-guide/Compaction.md
@@ -32,7 +32,7 @@ Suppose we ingest data from a Kafka broker, and we would like
to publish the dat
## Basic Usage
-`MRCompactor.compact()` is the entry point for MapReduce-based compaction. The
compaction unit is
[`Dataset`](https://github.com/linkedin/gobblin/blob/master/gobblin-compaction/src/main/java/gobblin/compaction/dataset/Dataset.java).
`MRCompactor` uses a
[`DatasetsFinder`](https://github.com/linkedin/gobblin/blob/master/gobblin-compaction/src/main/java/gobblin/compaction/dataset/DatasetsFinder.java)
to find all datasets eligible for compaction. Implementations of
`DatasetsFinder` include [...]
+`MRCompactor.compact()` is the entry point for MapReduce-based compaction. The
compaction unit is
[`Dataset`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/Dataset.java).
`MRCompactor` uses a
[`DatasetsFinder`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/dataset/DatasetsFinder.java)
to find all datasets eligible for compaction. Implem [...]
In the above example use case, for hourly compaction, each dataset contains an
hour's data in the `hourly_staging` folder, e.g.,
`/data/kafka_topics/PageViewEvent/hourly_staging/2015/10/29/08`; for daily
compaction, each dataset contains 24 hourly folder of a day, e.g.,
`/data/kafka_topics/PageViewEvent/hourly/2015/10/29`. In hourly compaction, you
may use the following config properties:
diff --git a/gobblin-docs/user-guide/Gobblin-Deployment.md
b/gobblin-docs/user-guide/Gobblin-Deployment.md
index 4b78387..00b8aef 100644
--- a/gobblin-docs/user-guide/Gobblin-Deployment.md
+++ b/gobblin-docs/user-guide/Gobblin-Deployment.md
@@ -7,7 +7,7 @@ Deployment Overview <a name="Standalone-Overview"></a>
--------------------
One important feature of Gobblin is that it can be run on different platforms.
Currently, Gobblin can run in standalone mode (which runs on a single machine),
and on Hadoop MapReduce mode (which runs on a Hadoop cluster). This page
summarizes the different deployment modes of Gobblin. It is important to
understand the architecture of Gobblin in a specific deployment mode, so this
page also describes the architecture of each deployment mode.
-Gobblin supports Java 7 and up, but can only run on Hadoop 2.x. By default,
Gobblin will build against Hadoop 2.x, run `./gradlew clean build`. More
information on how to build Gobblin can be found
[here](https://github.com/linkedin/gobblin/blob/master/README.md). All
directories/paths referred below are relative to `gobblin-dist`.
+Gobblin supports Java 7 and up, but can only run on Hadoop 2.x. By default,
Gobblin will build against Hadoop 2.x, run `./gradlew clean build`. More
information on how to build Gobblin can be found
[here](https://github.com/apache/incubator-gobblin/blob/master/README.md). All
directories/paths referred below are relative to `gobblin-dist`.
Standalone Architecture <a name="Standalone-Architecture"></a>
--------------------
diff --git a/gobblin-docs/user-guide/Gobblin-Schedulers.md
b/gobblin-docs/user-guide/Gobblin-Schedulers.md
index b0c1022..4166164 100644
--- a/gobblin-docs/user-guide/Gobblin-Schedulers.md
+++ b/gobblin-docs/user-guide/Gobblin-Schedulers.md
@@ -8,7 +8,7 @@ Gobblin jobs can be scheduled on a recurring basis using a few
different tools.
# Quartz
-Gobblin has a built in Quartz scheduler as part of the
[`JobScheduler`](https://github.com/linkedin/gobblin/blob/master/gobblin-scheduler/src/main/java/gobblin/scheduler/JobScheduler.java)
class. This class integrates with the Gobblin
[`SchedulerDaemon`](https://github.com/linkedin/gobblin/blob/master/gobblin-scheduler/src/main/java/gobblin/scheduler/SchedulerDaemon.java),
which can be run using the Gobblin
[`bin/gobblin-standalone.sh](https://github.com/linkedin/gobblin/blob/master/bin/
[...]
+Gobblin has a built in Quartz scheduler as part of the
[`JobScheduler`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/JobScheduler.java)
class. This class integrates with the Gobblin
[`SchedulerDaemon`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/scheduler/SchedulerDaemon.java),
which can be run using the Gobblin
[`bin/gobblin-standalone.sh](https://github.com [...]
So in order to take advantage of the Quartz scheduler two steps need to be
taken:
@@ -18,9 +18,9 @@ So in order to take advantage of the Quartz scheduler two
steps need to be taken
# Azkaban
-Gobblin can be launched via [Azkaban](https://azkaban.github.io/), and
open-source Workflow Manager for scheduling and launching Hadoop jobs.
Gobblin's
[`AzkabanJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-azkaban/src/main/java/gobblin/azkaban/AzkabanJobLauncher.java)
can be used to launch a Gobblin job through Azkaban.
+Gobblin can be launched via [Azkaban](https://azkaban.github.io/), and
open-source Workflow Manager for scheduling and launching Hadoop jobs.
Gobblin's
[`AzkabanJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/azkaban/AzkabanJobLauncher.java)
can be used to launch a Gobblin job through Azkaban.
-One has to follow the typical setup to create a zip file that can be uploaded
to Azkaban (it should include all dependent jars, which can be found in
`gobblin-dist.tar.gz`). The `.job` file for the Azkaban Job should contain all
configuration properties that would be put in a `.pull` file (for example, the
[Wikipedia
Example](https://github.com/linkedin/gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)
`.pull` file). All Gobblin system dependent properties (e.g. [`co [...]
+One has to follow the typical setup to create a zip file that can be uploaded
to Azkaban (it should include all dependent jars, which can be found in
`gobblin-dist.tar.gz`). The `.job` file for the Azkaban Job should contain all
configuration properties that would be put in a `.pull` file (for example, the
[Wikipedia
Example](https://github.com/apache/incubator-gobblin/blob/master/gobblin-example/src/main/resources/wikipedia.pull)
`.pull` file). All Gobblin system dependent properties (e [...]
In the Azkaban `.job` file, the `type` parameter should be set to `hadoopJava`
(see [here](http://azkaban.github.io/azkaban/docs/latest/#hadoopjava-type) for
more information about the `hadoopJava` Job Type). The `job.class` parameter
should be set to `gobblin.azkaban.AzkabanJobLauncher`.
@@ -38,13 +38,14 @@ This guide focuses on getting Gobblin to run in as a stand
alone Java Process. T
#### Example Config Files
-[`gobblin-oozie/src/main/resources/local`](https://github.com/linkedin/gobblin/tree/master/gobblin-oozie/src/main/resources/local)
contains sample configuration files for launching Gobblin Oozie. There are a
number of important files in this directory:
+[`gobblin-oozie/src/main/resources/local`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local)
contains sample configuration files for launching Gobblin Oozie. There are a
number of important files in this directory:
-`gobblin-oozie-example-system.properties` contains default system level
properties for Gobblin. When launched with Oozie, Gobblin will run inside a map
task; it is thus recommended to configure Gobblin to write directly to HDFS
rather than the local file system. The property `fs.uri` in this file should be
changed to point to the NameNode of the Hadoop File System the job should write
to. By default, all data is written under a folder called `gobblin-out`; to
change this modify the `gobb [...]
+[`gobblin-oozie-example-system.properties`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-system.properties)
contains default system level properties for Gobblin. When launched with
Oozie, Gobblin will run inside a map task; it is thus recommended to configure
Gobblin to write directly to HDFS rather than the local file system. The
property `fs.uri` in this file should be changed to point to the NameNode of
the Hadoop [...]
-`gobblin-oozie-example-workflow.properties` contains default Oozie properties
for any job launched. It is also the entry point for launching an Oozie job
(e.g. to launch an Oozie job from the command line you execute `oozie job
-config gobblin-oozie-example-workflow.properties -run`). In this file one
needs to update the `name.node` and `resource.manager` to the values specific
to their environment. Another important property in this file is
`oozie.wf.application.path`; it points to a fo [...]
+[`gobblin-oozie-example-workflow.properties`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-workflow.properties)
contains default Oozie properties for any job launched. It is also the entry
point for launching an Oozie job (e.g. to launch an Oozie job from the command
line you execute `oozie job -config gobblin-oozie-example-workflow.properties
-run`). In this file one needs to update the `name.node` and `resource.man [...]
-`gobblin-oozie-example-workflow.xml` contains an example Oozie workflow. This
example simply launches a Java process that invokes the main method of the
[`CliLocalJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/local/CliLocalJobLauncher.java).
The main method of this class expects two file paths to be passed to it (once
again these files need to be on HDFS). The `jobconfig` arg should point to a
file on HDFS containing all job c [...]
+[`gobblin-oozie-example-workflow.xml`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-oozie/src/test/resources/local/gobblin-oozie-example-workflow.xml)
contains an example Oozie workflow. This example simply launches a Java
process that invokes the main method of the
[`CliLocalJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/local/CliLocalJobLauncher.java).
The main method of this class expect [...]
+).
<!---Ying Do you think we can add some descriptions about launching through MR
mode? The simplest way is to use the <shell> tag and invoke
`gobblin-mapreduce.sh`. I've tested it before.-->
@@ -66,7 +67,7 @@ Launching Gobblin in mapreduce Mode works quite similar to
the local mode. In th
#### Example Config Files
-[`gobblin-oozie/src/main/resources/mapreduce`](https://github.com/linkedin/gobblin/tree/master/gobblin-oozie/src/main/resources/mapreduce)
contains sample configuration files for launching Gobblin Oozie in Mapreduce
mode. The main difference to launching Gobblin Oozie in Local mode are a view
extra MapReduce related configuration variables in the sysconfig.properties
file and launching CliMRJobLauncher instead CliLocalJobLauncher.
+[`gobblin-oozie/src/main/resources/mapreduce`](https://github.com/apache/incubator-gobblin/tree/master/gobblin-oozie/src/test/resources/mapreduce)
contains sample configuration files for launching Gobblin Oozie in Mapreduce
mode. The main difference to launching Gobblin Oozie in Local mode are a view
extra MapReduce related configuration variables in the sysconfig.properties
file and launching CliMRJobLauncher instead CliLocalJobLauncher.
#### Further steps
diff --git a/gobblin-docs/user-guide/Gobblin-on-Yarn.md
b/gobblin-docs/user-guide/Gobblin-on-Yarn.md
index 194cf4a..548f0f4 100644
--- a/gobblin-docs/user-guide/Gobblin-on-Yarn.md
+++ b/gobblin-docs/user-guide/Gobblin-on-Yarn.md
@@ -41,7 +41,7 @@ Heiix relies on ZooKeeper for its operations, and
particularly for maintaining t
## Gobblin Yarn Application Launcher
-The Gobblin Yarn Application Launcher (implemented by class
[`GobblinYarnAppLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinYarnAppLauncher.java))
is the client/driver of a Gobblin Yarn application. The first thing the
`GobblinYarnAppLauncher` does when it starts is to register itself with Helix
as a _spectator_ and creates a new Helix cluster with name specified through
the configuration property `gobblin.yarn.helix.cluster.name` [...]
+The Gobblin Yarn Application Launcher (implemented by class
[`GobblinYarnAppLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java))
is the client/driver of a Gobblin Yarn application. The first thing the
`GobblinYarnAppLauncher` does when it starts is to register itself with Helix
as a _spectator_ and creates a new Helix cluster with name specified through
the configuration property `gobblin.yarn. [...]
The `GobblinYarnAppLauncher` then sets up the Gobblin Yarn application and
submits it to run on Yarn. Once the Yarn application successfully starts
running, it starts an application state monitor that periodically checks the
state of the Gobblin Yarn application. If the state is one of the exit states
(`FINISHED`, `FAILED`, or `KILLED`), the `GobblinYarnAppLauncher` shuts down
itself.
@@ -49,19 +49,19 @@ Upon successfully submitting the application to run on
Yarn, the `GobblinYarnApp
### YarnAppSecurityManager
-The
[`YarnAppSecurityManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnAppSecurityManager.java)
works with the
[`YarnContainerSecurityManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnContainerSecurityManager.java)
running in the ApplicationMaster and the WorkUnitRunner for a complete
solution for security and delegation token management. The
`YarnAppSecurityManager` is responsible for [...]
+The
[`YarnAppSecurityManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnAppSecurityManager.java)
works with the
[`YarnContainerSecurityManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java)
running in the ApplicationMaster and the WorkUnitRunner for a complete
solution for security and delegation token management. The `Yarn [...]
### LogCopier
-The service
[`LogCopier`](https://github.com/linkedin/gobblin/blob/master/gobblin-utility/src/main/java/gobblin/util/logs/LogCopier.java)
in `GobblinYarnAppLauncher` streams the ApplicationMaster and WorkUnitRunner
logs in near real-time from the central location on HDFS where the logs are
streamed to from the ApplicationMaster and WorkUnitRunner containers, to the
local directory specified through the configuration property
`gobblin.yarn.logs.sink.root.dir` on the machine where the `Gob [...]
+The service
[`LogCopier`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java)
in `GobblinYarnAppLauncher` streams the ApplicationMaster and WorkUnitRunner
logs in near real-time from the central location on HDFS where the logs are
streamed to from the ApplicationMaster and WorkUnitRunner containers, to the
local directory specified through the configuration property
`gobblin.yarn.logs.sink.root.dir` on the mac [...]
## Gobblin ApplicationMaster
-The ApplicationMaster process runs the
[`GobblinApplicationMaster`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinApplicationMaster.java),
which uses a `ServiceManager` to manage the services supporting the operation
of the ApplicationMaster process. The services running in
`GobblinApplicationMaster` will be discussed later. When it starts, the first
thing `GobblinApplicationMaster` does is to connect to ZooKeeper and register
itself as a [...]
+The ApplicationMaster process runs the
[`GobblinApplicationMaster`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinApplicationMaster.java),
which uses a `ServiceManager` to manage the services supporting the operation
of the ApplicationMaster process. The services running in
`GobblinApplicationMaster` will be discussed later. When it starts, the first
thing `GobblinApplicationMaster` does is to connect to ZooKeeper and re [...]
### YarnService
-The service
[`YarnService`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnService.java)
handles all Yarn-related task including the following:
+The service
[`YarnService`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnService.java)
handles all Yarn-related task including the following:
* Registering and un-registering the ApplicationMaster with the Yarn
ResourceManager.
* Requesting the initial set of containers from the Yarn ResourceManager.
@@ -71,37 +71,37 @@ This design makes it switch to a different resource
manager, e.g., Mesos, by rep
### GobblinHelixJobScheduler
-[`GobblinApplicationMaster`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinApplicationMaster.java)
runs the
[`GobblinHelixJobScheduler`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinHelixJobScheduler.java)
that schedules jobs to run through the Helix [Distributed Task Execution
Framework](http://helix.apache.org/0.7.1-docs/recipes/task_dag_execution.html).
For each Gobblin job run, the `Gob [...]
+[`GobblinApplicationMaster`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinApplicationMaster.java)
runs the
[`GobblinHelixJobScheduler`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobScheduler.java)
that schedules jobs to run through the Helix [Distributed Task Execution
Framework](http://helix.apache.org/0.7.1-docs/recipes/task_dag_executi [...]
-Like the
[`LocalJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/local/LocalJobLauncher.java)
and
[`MRJobLauncher`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/mapreduce/MRJobLauncher.java),
the `GobblinHelixJobLauncher` handles output data commit and job state
persistence.
+Like the
[`LocalJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/local/LocalJobLauncher.java)
and
[`MRJobLauncher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/mapreduce/MRJobLauncher.java),
the `GobblinHelixJobLauncher` handles output data commit and job state
persistence.
### LogCopier
-The service
[`LogCopier`](https://github.com/linkedin/gobblin/blob/master/gobblin-utility/src/main/java/gobblin/util/logs/LogCopier.java)
in `GobblinApplicationMaster` streams the ApplicationMaster logs in near
real-time from the machine running the ApplicationMaster container to a central
location on HDFS so the logs can be accessed at runtime. More details on this
can be found in [Log Aggregation](#log-aggregation).
+The service
[`LogCopier`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java)
in `GobblinApplicationMaster` streams the ApplicationMaster logs in near
real-time from the machine running the ApplicationMaster container to a central
location on HDFS so the logs can be accessed at runtime. More details on this
can be found in [Log Aggregation](#log-aggregation).
### YarnContainerSecurityManager
-The
[`YarnContainerSecurityManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnContainerSecurityManager.java)
runs in both the ApplicationMaster and the WorkUnitRunner. When it starts, it
registers a message handler with the `HelixManager` for handling messages on
refreshes of the delegation token. Once such a message is received, the
`YarnContainerSecurityManager` gets the path to the token file on HDFS from the
message, and updated the [...]
+The
[`YarnContainerSecurityManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java)
runs in both the ApplicationMaster and the WorkUnitRunner. When it starts, it
registers a message handler with the `HelixManager` for handling messages on
refreshes of the delegation token. Once such a message is received, the
`YarnContainerSecurityManager` gets the path to the token file on HDFS from the
messag [...]
## Gobblin WorkUnitRunner
-The WorkUnitRunner process runs the
[`GobblinWorkUnitRunner`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinWorkUnitRunner.java),
which uses a `ServiceManager` to manage the services supporting the operation
of the WorkUnitRunner process. The services running in `GobblinWorkUnitRunner`
will be discussed later. When it starts, the first thing
`GobblinWorkUnitRunner` does is to connect to ZooKeeper and register itself as
a Helix _participant [...]
+The WorkUnitRunner process runs as part of
[`GobblinTaskRunner`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinTaskRunner.java),
which uses a `ServiceManager` to manage the services supporting the operation
of the WorkUnitRunner process. The services running in `GobblinWorkUnitRunner`
will be discussed later. When it starts, the first thing
`GobblinWorkUnitRunner` does is to connect to ZooKeeper and register itself
[...]
### TaskExecutor
-The
[`TaskExecutor`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/TaskExecutor.java)
remains the same as in the standalone and MR modes, and is purely responsible
for running tasks assigned to a WorkUnitRunner.
+The
[`TaskExecutor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskExecutor.java)
remains the same as in the standalone and MR modes, and is purely responsible
for running tasks assigned to a WorkUnitRunner.
### GobblinHelixTaskStateTracker
-The
[`GobblinHelixTaskStateTracker`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/GobblinHelixTaskStateTracker.java)
has a similar responsibility as the `LocalTaskStateTracker` and
`MRTaskStateTracker`: keeping track of the state of running tasks including
operational metrics, e.g., total records pulled, records pulled per second,
total bytes pulled, bytes pulled per second, etc.
+The
[`GobblinHelixTaskStateTracker`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixTaskStateTracker.java)
has a similar responsibility as the `LocalTaskStateTracker` and
`MRTaskStateTracker`: keeping track of the state of running tasks including
operational metrics, e.g., total records pulled, records pulled per second,
total bytes pulled, bytes pulled per second, etc.
### LogCopier
-The service
[`LogCopier`](https://github.com/linkedin/gobblin/blob/master/gobblin-utility/src/main/java/gobblin/util/logs/LogCopier.java)
in `GobblinWorkUnitRunner` streams the WorkUnitRunner logs in near real-time
from the machine running the WorkUnitRunner container to a central location on
HDFS so the logs can be accessed at runtime. More details on this can be found
in [Log Aggregation](#log-aggregation).
+The service
[`LogCopier`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-utility/src/main/java/org/apache/gobblin/util/logs/LogCopier.java)
in `GobblinWorkUnitRunner` streams the WorkUnitRunner logs in near real-time
from the machine running the WorkUnitRunner container to a central location on
HDFS so the logs can be accessed at runtime. More details on this can be found
in [Log Aggregation](#log-aggregation).
### YarnContainerSecurityManager
-The
[`YarnContainerSecurityManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/YarnContainerSecurityManager.java)
in `GobblinWorkUnitRunner` works in the same way as it in
`GobblinApplicationMaster`.
+The
[`YarnContainerSecurityManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnContainerSecurityManager.java)
in `GobblinWorkUnitRunner` works in the same way as it in
`GobblinApplicationMaster`.
## Failure Handling
@@ -305,8 +305,8 @@ Gobblin on Yarn is backward compatible and supports
existing Gobblin jobs runnin
# Monitoring
-Gobblin Yarn uses the [Gobblin Metrics](../metrics/Gobblin-Metrics) library
for collecting and reporting metrics at the container, job, and task levels.
Each `GobblinWorkUnitRunner` maintains a
[`ContainerMetrics`](https://github.com/linkedin/gobblin/blob/master/gobblin-yarn/src/main/java/gobblin/yarn/ContainerMetrics.java)
that is the parent of the
[`JobMetrics`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/util/JobMetrics.java)
of each j [...]
+Gobblin Yarn uses the [Gobblin Metrics](../metrics/Gobblin-Metrics) library
for collecting and reporting metrics at the container, job, and task levels.
Each `GobblinWorkUnitRunner` maintains a
[`ContainerMetrics`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-cluster/src/main/java/org/apache/gobblin/cluster/ContainerMetrics.java)
that is the parent of the
[`JobMetrics`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gob
[...]
Collected metrics can be reported to various sinks such as Kafka, files, and
JMX, depending on the configuration. Specifically, `metrics.enabled` controls
whether metrics collecting and reporting are enabled or not.
`metrics.reporting.kafka.enabled`, `metrics.reporting.file.enabled`, and
`metrics.reporting.jmx.enabled` control whether collected metrics should be
reported or not to Kafka, files, and JMX, respectively. Please refer to
[Metrics Properties](Configuration-Properties-Glossary# [...]
-In addition to metric collecting and reporting, Gobblin Yarn also supports
writing job execution information to a MySQL-backed job execution history
store, which keeps track of job execution information. Please refer to the
[DDL](https://github.com/linkedin/gobblin/blob/master/gobblin-metastore/src/main/resources/gobblin_job_history_store.sql)
for the relevant MySQL tables. Detailed information on the job execution
history store including how to configure it can be found [here](Job-Execu [...]
+In addition to metric collecting and reporting, Gobblin Yarn also supports
writing job execution information to a MySQL-backed job execution history
store, which keeps track of job execution information. Please refer to the
[DDL](https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration)
for the relevant MySQL tables. Detailed information on the job execution
history store including how to configure it can be found
[here](Job-Execution-Hist [...]
diff --git a/gobblin-docs/user-guide/Hive-Registration.md
b/gobblin-docs/user-guide/Hive-Registration.md
index c9cb6bb..5751e50 100644
--- a/gobblin-docs/user-guide/Hive-Registration.md
+++ b/gobblin-docs/user-guide/Hive-Registration.md
@@ -6,17 +6,17 @@ Gobblin has the ability to register the ingested/compacted
data in Hive. This al
## How Hive Registration Works in Gobblin
-Hive registration is done in
[`HiveRegister`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/HiveRegister.java).
After the data is published, the publisher or compaction runner will create an
instance of `HiveRegister`, and for each path that should be registered in
Hive, the publisher or compaction runner will use a specific
[`HiveRegistrationPolicy`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/
[...]
+Hive registration is done in
[`HiveRegister`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveRegister.java).
After the data is published, the publisher or compaction runner will create an
instance of `HiveRegister`, and for each path that should be registered in
Hive, the publisher or compaction runner will use a specific
[`HiveRegistrationPolicy`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-
[...]
### `HiveSpec`
-A `HiveSpec` specifies how a path should be registered in Hive, i.e., which
database, which table, which partition should the path be registered. An
example is
[`SimpleHiveSpec`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/spec/SimpleHiveSpec.java).
+A `HiveSpec` specifies how a path should be registered in Hive, i.e., which
database, which table, which partition should the path be registered. An
example is
[`SimpleHiveSpec`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/SimpleHiveSpec.java).
### `HiveRegistrationPolicy`
`HiveRegistrationPolicy` is responsible for generating `HiveSpec`s given a
path. For example, if you want paths ending with a date (e.g.,
`/(something)/2016/05/22`) to be registered in the corresponding daily
partition (e.g., `daily-2016-05-22`), you can create an implementation of
`HiveRegistrationPolicy` that contains the logic of converting such a path into
a Hive partition.
-An example is
[`HiveRegistrationPolicyBase`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/policy/HiveRegistrationPolicyBase.java),
which provides base implementation for getting database names and table names
for a path:
+An example is
[`HiveRegistrationPolicyBase`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/policy/HiveRegistrationPolicyBase.java),
which provides base implementation for getting database names and table names
for a path:
* A database/table name can be specified explicitly in `hive.database.name` or
`hive.table.name`.
* Alternatively, a database/table regex can be provided in
`hive.database.regex` or `hive.table.regex`. The regex will be matched against
the path to be registered, and if they match, the first group is considered the
database/table name.
@@ -27,13 +27,13 @@ One should in general extend `HiveRegistrationPolicyBase`
when implementing a ne
### `HiveSerDeManager`
-If the data to be registered is in a format other than plain text (CSV, TSV,
etc.), you often need to use a SerDe and specify some SerDe properties
including the type of SerDe, input format, output format, schema, etc. This is
done in
[`HiveSerDeManager`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/HiveSerDeManager.java),
which is part of a
[`HiveRegistrationUnit`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-regist
[...]
+If the data to be registered is in a format other than plain text (CSV, TSV,
etc.), you often need to use a SerDe and specify some SerDe properties
including the type of SerDe, input format, output format, schema, etc. This is
done in
[`HiveSerDeManager`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/HiveSerDeManager.java),
which is part of a
[`HiveRegistrationUnit`](https://github.com/apache/incubator-gobblin/blob [...]
### Predicate and Activity
-One or more `Predicate`s can be attached to a `HiveSpec`. If a `HiveSpec`
contains `Predicate`s, unless `Predicate`s return `true`, the `HiveSpec` will
not be registered. This is useful in cases where, for example, one only wants
to register a path in Hive if a particular Hive table or partition doesn't
already exist. An example is
[`TableNotExistPredicate`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/spec/predicate/TableNotExistPr
[...]
+One or more `Predicate`s can be attached to a `HiveSpec`. If a `HiveSpec`
contains `Predicate`s, unless `Predicate`s return `true`, the `HiveSpec` will
not be registered. This is useful in cases where, for example, one only wants
to register a path in Hive if a particular Hive table or partition doesn't
already exist. An example is
[`TableNotExistPredicate`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/predic
[...]
-One or more
[`Activity`](https://github.com/linkedin/gobblin/blob/master/gobblin-hive-registration/src/main/java/gobblin/hive/spec/activity/Activity.java)s
can be attached to a `HiveSpec`. There are two types of activities:
pre-activities and post-activities, which will be executed before and after a
`HiveSpec` is registered, respectively. This is useful, for example, when you
need to drop/alter a table/partition before or after a path is registered. An
example is [`DropTableActivity`](h [...]
+One or more
[`Activity`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/spec/activity/Activity.java)s
can be attached to a `HiveSpec`. There are two types of activities:
pre-activities and post-activities, which will be executed before and after a
`HiveSpec` is registered, respectively. This is useful, for example, when you
need to drop/alter a table/partition before or after a path is registered. An
example is [`Dr [...]
## How to Use Hive Registration in Your Gobblin Job
@@ -46,10 +46,10 @@ Example table/partition properties are "owner" and
"retention", example storage
If you are running a Gobblin ingestion job:
-* If data is published in the job (which is the default case), use a job-level
data publisher that can perform Hive registration, such as
[`BaseDataPublisherWithHiveRegistration`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/publisher/BaseDataPublisherWithHiveRegistration.java).
If you need to do Hive registration with a different publisher than
`BaseDataPublisher`, you will need to extend that publisher to do Hive
registration, which will be similar [...]
-* If data is published in the tasks, use
[`HiveRegistrationPublisher`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/publisher/HiveRegistrationPublisher.java)
as the job-level data publisher. This publisher does not publish any data; it
only does Hive registration.
+* If data is published in the job (which is the default case), use a job-level
data publisher that can perform Hive registration, such as
[`BaseDataPublisherWithHiveRegistration`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/BaseDataPublisherWithHiveRegistration.java).
If you need to do Hive registration with a different publisher than
`BaseDataPublisher`, you will need to extend that publisher to do Hive
registration, wh [...]
+* If data is published in the tasks, use
[`HiveRegistrationPublisher`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/publisher/HiveRegistrationPublisher.java)
as the job-level data publisher. This publisher does not publish any data; it
only does Hive registration.
-If you are running a Gobblin compaction job: add
[`HiveRegistrationCompactorListener`](https://github.com/linkedin/gobblin/blob/master/gobblin-compaction/src/main/java/gobblin/compaction/hive/registration/HiveRegistrationCompactorListener.java)
to the list of compaction listeners by adding the class name to property
`compaction.listeners`.
+If you are running a Gobblin compaction job: add
[`HiveRegistrationCompactorListener`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-compaction/src/main/java/org/apache/gobblin/compaction/hive/registration/HiveRegistrationCompactorListener.java)
to the list of compaction listeners by adding the class name to property
`compaction.listeners`.
## Hive Registration Config Properties
diff --git a/gobblin-docs/user-guide/Job-Execution-History-Store.md
b/gobblin-docs/user-guide/Job-Execution-History-Store.md
index 04357a0..cbf3d01 100644
--- a/gobblin-docs/user-guide/Job-Execution-History-Store.md
+++ b/gobblin-docs/user-guide/Job-Execution-History-Store.md
@@ -54,7 +54,7 @@ The following table summarizes task-level execution
information the Job Executio
Default Implementation
--------------------------------
-The default implementation of the Job Execution History Store stores job
execution information into a MySQL database in a few different tables.
Specifically, the following tables are used and should be created before
writing to the store is enabled. Checkout the MySQL
[DDLs](https://github.com/linkedin/gobblin/blob/master/gobblin-metastore/src/main/resources/gobblin_job_history_store.sql)
of the tables for detailed columns of each table.
+The default implementation of the Job Execution History Store stores job
execution information into a MySQL database in a few different tables.
Specifically, the following tables are used and should be created before
writing to the store is enabled. Checkout the MySQL
[DDLs](https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration)
of the tables for detailed columns of each table.
* Table `gobblin_job_executions` stores basic information about a job
execution including the start and end times, job running state, number of
launched and completed tasks, etc.
* Table `gobblin_task_executions` stores basic information on task executions
of a job, including the start and end times, task running state, task failure
message if any, etc, of each task.
diff --git a/gobblin-docs/user-guide/Monitoring.md
b/gobblin-docs/user-guide/Monitoring.md
index 7db9fc0..c70b103 100644
--- a/gobblin-docs/user-guide/Monitoring.md
+++ b/gobblin-docs/user-guide/Monitoring.md
@@ -42,7 +42,7 @@ Among the above metrics, `${metric_group}.${id}.records` and
`${metric_group}.${
Job Execution History Store
--------------------
-Gobblin also supports writing job execution information to a job execution
history store backed by a database of choice. Gobblin uses MySQL by default and
it ships with the SQL
[DDLs](https://github.com/linkedin/gobblin/wiki/files/gobblin_job_history_store_ddlwq.sql)
of the relevant MySQL tables, although it still allows users to choose which
database to use as long as the schema of the tables is compatible. Users can
use the properties `job.history.store.url` and `job.history.store.jdb [...]
+Gobblin also supports writing job execution information to a job execution
history store backed by a database of choice. Gobblin uses MySQL by default and
it ships with the SQL
[DDLs](https://github.com/apache/incubator-gobblin/tree/master/gobblin-metastore/src/main/resources/db/migration)
of the relevant MySQL tables, although it still allows users to choose which
database to use as long as the schema of the tables is compatible. Users can
use the properties `job.history.store.url` and [...]
```
job.history.store.url=jdbc:mysql://localhost/gobblin
diff --git a/gobblin-docs/user-guide/State-Management-and-Watermarks.md
b/gobblin-docs/user-guide/State-Management-and-Watermarks.md
index 2e53918..da5b5ea 100644
--- a/gobblin-docs/user-guide/State-Management-and-Watermarks.md
+++ b/gobblin-docs/user-guide/State-Management-and-Watermarks.md
@@ -23,7 +23,7 @@ In the next run, the `Source` will call
`SourceState.getPreviousWorkUnitStates()
**watermark type**
-A watermark can be of any custom type by implementing the
[`Watermark`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/source/extractor/Watermark.java)
interface. For example, for Kafka-HDFS ingestion, if each `WorkUnit` is
responsible for pulling a single Kafka topic partition, a watermark is a single
`long` value representing a Kafka offset. If each `WorkUnit` is responsible for
pulling multiple Kafka topic partitions, a watermark can be a list of `lo [...]
+A watermark can be of any custom type by implementing the
[`Watermark`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/source/extractor/Watermark.java)
interface. For example, for Kafka-HDFS ingestion, if each `WorkUnit` is
responsible for pulling a single Kafka topic partition, a watermark is a single
`long` value representing a Kafka offset. If each `WorkUnit` is responsible for
pulling multiple Kafka topic partitions, a watermark c [...]
### Task Failures
@@ -31,7 +31,7 @@ A task may pull some data and then fail. If a task fails and
job commit policy s
### Multi-Dataset Jobs
-Currently the only state store implementation Gobblin provides is
[`FsStateStore`](https://github.com/linkedin/gobblin/blob/master/gobblin-metastore/src/main/java/gobblin/metastore/FsStateStore.java)
which uses Hadoop SequenceFiles to store the states. By default, each job run
reads the SequenceFile created by the previous run, and generates a new
SequenceFile. This creates a pitfall when a job pulls data from multiple
datasets: if a data set is skipped in a job run for whatever reason ( [...]
+Currently the only state store implementation Gobblin provides is
[`FsStateStore`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java)
which uses Hadoop SequenceFiles to store the states. By default, each job run
reads the SequenceFile created by the previous run, and generates a new
SequenceFile. This creates a pitfall when a job pulls data from multiple
datasets: if a data set is skipped in a job run fo [...]
**Example**: suppose we schedule a Gobblin job to pull a Kafka topic from a
Kafka broker, which has 10 partitions. In this case each partition is a
dataset. In one of the job runs, a partition is skipped due to either being
blacklisted or some failure. If no `WorkUnit` is created for this partition,
this partition's watermark will not be checked in to the state store, and will
not be available for the next run.
@@ -47,7 +47,7 @@ Note that when using Dataset URNs, **each `WorkUnit` can only
have one `dataset.
## Gobblin State Deep Dive
-Gobblin involves several types of states during a job run, such as `JobState`,
`TaskState`, `WorkUnit`, etc. They all extend the
[`State`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/configuration/State.java)
class, which is a wrapper around
[`Properties`](https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html)
and provides some useful utility functions.
+Gobblin involves several types of states during a job run, such as `JobState`,
`TaskState`, `WorkUnit`, etc. They all extend the
[`State`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/configuration/State.java)
class, which is a wrapper around
[`Properties`](https://docs.oracle.com/javase/8/docs/api/java/util/Properties.html)
and provides some useful utility functions.
### `State` class hierarchy
@@ -78,7 +78,7 @@ The `MultiWorkUnit` is useful for finer-grained control and
load balancing. With
* The job launcher then passes the `JobState` (as a `SourceState` object) to
the `Source`, based on which the `Source` will create a set of `WorkUnit`s.
Note that when creating `WorkUnit`s, the `Source` should not add properties in
`SourceState` into the `WorkUnit`s, which will be done when each `WorkUnit` is
executed in a `Task`. The reason is that since the job launcher runs in a
single JVM, creating a large number of `WorkUnit`s, each containing a copy of
the `SourceState`, may cause OOM.
* The job launcher prepares to run the `WorkUnit`s.
- * In standalone mode, the job launcher will add properties in the `JobState`
into each `WorkUnit` (if a property in `JobState` already exists in the
`WorkUnit`, it will NOT be overwritten, i.e., the value in the `WorkUnit` takes
precedence). Then for each `WorkUnit` it creates a `Task` to run the
`WorkUnit`, and submits all these Tasks to a
[`TaskExecutor`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/TaskExecutor.java)
which will run the [...]
+ * In standalone mode, the job launcher will add properties in the `JobState`
into each `WorkUnit` (if a property in `JobState` already exists in the
`WorkUnit`, it will NOT be overwritten, i.e., the value in the `WorkUnit` takes
precedence). Then for each `WorkUnit` it creates a `Task` to run the
`WorkUnit`, and submits all these Tasks to a
[`TaskExecutor`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/TaskExecutor.java)
[...]
* In MR mode, the job launcher will serialize the `JobState` and each
`WorkUnit` into a file, which will be picked up by the mappers. It then
creates, configures and submits a Hadoop job.
After this step, the job launcher will be waiting till all tasks finish.
diff --git a/gobblin-docs/user-guide/Working-with-Job-Configuration-Files.md
b/gobblin-docs/user-guide/Working-with-Job-Configuration-Files.md
index 87aaa19..b4f3d42 100644
--- a/gobblin-docs/user-guide/Working-with-Job-Configuration-Files.md
+++ b/gobblin-docs/user-guide/Working-with-Job-Configuration-Files.md
@@ -5,7 +5,7 @@ Table of Contents
Job Configuration Basics
--------------------
-A Job configuration file is a text file with extension `.pull` or `.job` that
defines the job properties that can be loaded into a Java
[Properties](http://docs.oracle.com/javase/7/docs/api/java/util/Properties.html)
object. Gobblin uses
[commons-configuration](http://commons.apache.org/proper/commons-configuration/)
to allow variable substitutions in job configuration files. You can find some
example Gobblin job configuration files
[here](https://github.com/linkedin/gobblin/tree/master/ [...]
+A Job configuration file is a text file with extension `.pull` or `.job` that
defines the job properties that can be loaded into a Java
[Properties](http://docs.oracle.com/javase/7/docs/api/java/util/Properties.html)
object. Gobblin uses
[commons-configuration](http://commons.apache.org/proper/commons-configuration/)
to allow variable substitutions in job configuration files. You can find some
example Gobblin job configuration files
[here](https://github.com/apache/incubator-gobblin/tree [...]
A Job configuration file typically includes the following properties, in
additional to any mandatory configuration properties required by the custom
[Gobblin Constructs](Gobblin-Architecture#gobblin-constructs) classes. For a
complete reference of all configuration properties supported by Gobblin, please
refer to [Configuration Properties Glossary](Configuration-Properties-Glossary).
diff --git a/gobblin-docs/user-guide/Working-with-the-ForkOperator.md
b/gobblin-docs/user-guide/Working-with-the-ForkOperator.md
index 40bf854..2416c9e 100644
--- a/gobblin-docs/user-guide/Working-with-the-ForkOperator.md
+++ b/gobblin-docs/user-guide/Working-with-the-ForkOperator.md
@@ -6,7 +6,7 @@ Table of Contents
Overview of the ForkOperator
--------------------
-The
[`ForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/fork/ForkOperator.java)
is a type of control operators that allow a task flow to branch into multiple
streams (or forked branches) as represented by a
[`Fork`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/Fork.java),
each of which goes to a separately configured sink with its own data writer.
The `ForkOperator` gives users more flexibilit [...]
+The
[`ForkOperator`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java)
is a type of control operators that allow a task flow to branch into multiple
streams (or forked branches) as represented by a
[`Fork`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java),
each of which goes to a separately configured sink with its own data writer.
The [...]
<p align="center">
<figure>
@@ -20,21 +20,21 @@ Using the ForkOperator
### Basics of Usage
-The
[`ForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/fork/ForkOperator.java),
like most other operators in a Gobblin task flow, is pluggable through the
configuration, or more specifically , the configuration property
`fork.operator.class` that points to a class that implements the `ForkOperator`
interface. For instance:
+The
[`ForkOperator`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/ForkOperator.java),
like most other operators in a Gobblin task flow, is pluggable through the
configuration, or more specifically , the configuration property
`fork.operator.class` that points to a class that implements the `ForkOperator`
interface. For instance:
```
fork.operator.class=org.apache.gobblin.fork.IdentityForkOperator
```
-By default, if no `ForkOperator` class is specified, internally Gobblin uses
the default implementation
[`IdentityForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/fork/IdentityForkOperator.java)
with a single forked branch (although it does supports multiple forked
branches). The `IdentityForkOperator` simply unconditionally forwards the
schema and ingested data records to all the forked branches, the number of
which is specified through t [...]
+By default, if no `ForkOperator` class is specified, internally Gobblin uses
the default implementation
[`IdentityForkOperator`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/IdentityForkOperator.java)
with a single forked branch (although it does supports multiple forked
branches). The `IdentityForkOperator` simply unconditionally forwards the
schema and ingested data records to all the forked branches, the number of
which is [...]
The _expected_ number of forked branches is given by the method `getBranches`
of the `ForkOperator`. This number must match the size of the list of
`Boolean`s returned by `forkSchema` as well as the size of the list of
`Boolean`s returned by `forkDataRecords`. Otherwise,
`ForkBranchMismatchException` will be thrown. Note that the `ForkOperator`
itself _is not making and returning a copy_ for the input schema and data
records, but rather just providing a `Boolean` for each forked branch t [...]
The use of the `ForkOperator` with _the possibility that the schema and/or
data records may be forwarded to more than one forked branches_ has some
special requirement on the input schema and data records to the `ForkOperator`.
Specifically, because the same schema or data records may be forwarded to more
than branches that may alter the schema or data records in place, it is
necessary for the Gobblin task flow to make a copy of the input schema or data
records for each forked branch so [...]
-To guarantee that it is always able to make a copy in such a case, Gobblin
requires the input schema and data records to be of type `Copyable` when there
are more than one forked branch.
[`Copyable`](https://github.com/linkedin/gobblin/blob/master/gobblin-api/src/main/java/gobblin/fork/Copyable.java)
is an interface that defines a method `copy` for making a copy of an instance
of a given type. The Gobblin task flow will check if the input schema and data
records are instances of `Copyabl [...]
+To guarantee that it is always able to make a copy in such a case, Gobblin
requires the input schema and data records to be of type `Copyable` when there
are more than one forked branch.
[`Copyable`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-api/src/main/java/org/apache/gobblin/fork/Copyable.java)
is an interface that defines a method `copy` for making a copy of an instance
of a given type. The Gobblin task flow will check if the input schema and data
records are in [...]
-Gobblin ships with some built-in `Copyable` implementations, e.g.,
[`CopyableSchema`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/fork/CopyableSchema.java)
and
[`CopyableGenericRecord`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/fork/CopyableGenericRecord.java)
for Avro's `Schema` and `GenericRecord`.
+Gobblin ships with some built-in `Copyable` implementations, e.g.,
[`CopyableSchema`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/CopyableSchema.java)
and
[`CopyableGenericRecord`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/CopyableGenericRecord.java)
for Avro's `Schema` and `GenericRecord`.
### Per-Fork Configuration
@@ -51,15 +51,15 @@ In a normal task flow where the default
`IdentityForkOperator` with a single bra
* The failure of any forked branch means the failure of the whole task flow,
i.e., the task succeeds if and only if all the forked branches succeed.
* A forked branch stops processing any outstanding incoming data records in
the queue if it fails in the middle of processing the data.
* The failure and subsequent stop/completion of any forked branch does not
prevent other branches from processing their copies of the ingested data
records. The task will wait until all the branches to finish, regardless if
they succeed or fail.
-* The commit of output data of forks is determined by the job commit policy
(see
[`JobCommitPolicy`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/source/extractor/JobCommitPolicy.java))
specified. If `JobCommitPolicy.COMMIT_ON_FULL_SUCCESS` (or `full` in short) is
used, the output data of the entire job will be discarded if any forked branch
fails, which will fail the task and consequently the job. If instead
`JobCommitPolicy.COMMIT_SUCCESSFUL_TASKS` [...]
+* The commit of output data of forks is determined by the job commit policy
(see
[`JobCommitPolicy`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core-base/src/main/java/org/apache/gobblin/source/extractor/JobCommitPolicy.java))
specified. If `JobCommitPolicy.COMMIT_ON_FULL_SUCCESS` (or `full` in short) is
used, the output data of the entire job will be discarded if any forked branch
fails, which will fail the task and consequently the job. If instead
`JobCommitPolicy. [...]
### Performance Tuning
-Internally, each forked branch as represented by a
[`Fork`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/Fork.java)
maintains a bounded record queue (implemented by
[`BoundedBlockingRecordQueue`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/BoundedBlockingRecordQueue.java)),
which serves as a buffer between the pre-fork stream and the forked stream of
the particular branch. The size if this [...]
+Internally, each forked branch as represented by a
[`Fork`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java)
maintains a bounded record queue (implemented by
[`BoundedBlockingRecordQueue`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/BoundedBlockingRecordQueue.java)),
which serves as a buffer between the pre-fork stream and the forked stream
[...]
In terms of the number of forked branches, we have seen use cases with a half
dozen forked branches, and we are anticipating uses cases with much larger
numbers. Again, when using a large number of forked branches, the size of the
record queues and the timeout time need to be carefully tuned.
-The
[`BoundedBlockingRecordQueue`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/BoundedBlockingRecordQueue.java)
in each
[`Fork`](https://github.com/linkedin/gobblin/blob/master/gobblin-runtime/src/main/java/gobblin/runtime/Fork.java)
keeps trach of the following queue statistics that can be output to the logs
if the `DEBUG` logging level is turned on. Those statistics provide good
indications on the performance of the forks.
+The
[`BoundedBlockingRecordQueue`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/BoundedBlockingRecordQueue.java)
in each
[`Fork`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-runtime/src/main/java/org/apache/gobblin/runtime/fork/Fork.java)
keeps trach of the following queue statistics that can be output to the logs
if the `DEBUG` logging level is turned on. Those statistics provide good
indications on [...]
* Queue size, i.e., the number of records in queue.
* Queue fill ratio, i.e., a ratio of the number of records in queue over the
queue capacity.
@@ -70,7 +70,7 @@ The
[`BoundedBlockingRecordQueue`](https://github.com/linkedin/gobblin/blob/mast
### Comparison with PartitionedDataWriter
-Gobblin ships with a special type of `DataWriter`s called
[`PartitionedDataWriter`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/writer/PartitionedDataWriter.java)
that allow ingested records to be written in a partitioned fashion using a
`WriterPartitioner` into different locations in the same sink. The
`WriterPartitioner` determines the specific partition for each data record. So
there's certain overlap in terms of functionality between the `ForkOp [...]
+Gobblin ships with a special type of `DataWriter`s called
[`PartitionedDataWriter`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/writer/PartitionedDataWriter.java)
that allow ingested records to be written in a partitioned fashion using a
`WriterPartitioner` into different locations in the same sink. The
`WriterPartitioner` determines the specific partition for each data record. So
there's certain overlap in terms of functionality [...]
* The `ForkOperator` requires the number of forked branches to be known and
returned through `getBranches` before the task starts, whereas the
`PartitionedDataWriter` does not have this requirement.
* The `PartitionedDataWriter` writes each data record to a single partition,
whereas the `ForkOperator` allows data records to be forwarded to any number of
forked branches.
@@ -81,7 +81,7 @@ Gobblin ships with a special type of `DataWriter`s called
[`PartitionedDataWrite
Writing your Own ForkOperator
--------------------
-Since the built-in default implementation
[`IdentityForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin-core/src/main/java/gobblin/fork/IdentityForkOperator.java)
simply blindly forks the input schema and data records to every branches, it's
often necessary to have a custom implementation of the `ForkOperator` interface
for more fine-grained control on the actual branching. Checkout the interface
[`ForkOperator`](https://github.com/linkedin/gobblin/blob/master/gobblin- [...]
+Since the built-in default implementation
[`IdentityForkOperator`](https://github.com/apache/incubator-gobblin/blob/master/gobblin-core/src/main/java/org/apache/gobblin/fork/IdentityForkOperator.java)
simply blindly forks the input schema and data records to every branches, it's
often necessary to have a custom implementation of the `ForkOperator` interface
for more fine-grained control on the actual branching. Checkout the interface
[`ForkOperator`](https://github.com/apache/incubator-g [...]
Best Practices
--------------------
diff --git a/gradle/scripts/computeVersions.gradle
b/gradle/scripts/computeVersions.gradle
index cf7baaa..10fdf25 100644
--- a/gradle/scripts/computeVersions.gradle
+++ b/gradle/scripts/computeVersions.gradle
@@ -18,7 +18,7 @@
if (!project.hasProperty('version') || project.version == 'unspecified') {
try {
exec {
- commandLine 'git', 'fetch', '-t',
'https://github.com/linkedin/gobblin.git', 'master'
+ commandLine 'git', 'fetch', '-t',
'https://github.com/apache/incubator-gobblin.git', 'master'
}
def versionOut = new ByteArrayOutputStream()
exec {
diff --git a/mkdocs.yml b/mkdocs.yml
index b6a77f9..5c4a345 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -2,7 +2,7 @@ site_name: Gobblin Documentation
site_description: Documentation for Gobblin, a universal data ingestion
framework for Hadoop
site_author: Abhishek Tiwari, Aditya Sharma, Chavdar Botev, Issac Buenrostro,
Min Tu, Narasimha Veeramreedy, Pradhan Cadabam, Raul Agepati, Sahil Takiar,
Shirshanka Das, Yinan Li, Ying Dai, Ziyang Liu
-repo_url: https://github.com/linkedin/gobblin
+repo_url: https://github.com/apache/incubator-gobblin
repo_name: Gobblin
docs_dir: gobblin-docs
