[
https://issues.apache.org/jira/browse/GOBBLIN-1521?focusedWorklogId=641238&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641238
]
ASF GitHub Bot logged work on GOBBLIN-1521:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 24/Aug/21 17:58
Start Date: 24/Aug/21 17:58
Worklog Time Spent: 10m
Work Description: sv2000 commented on a change in pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#discussion_r695069713
##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming
manner. Normally it's running on yarn, to get the full function,
+you need have helix cluster, yarn cluster, kafka server and schema registry
set up. But we also have one local mode of streaming task using EmbeddedGobblin
+for user to easily start and test out the job. Here is the steps:
+
+# Setup local kafka cluster
+
+Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up
your kafka cluster, and create test topic "testEvents"
+
+# Run EmbeddedGobblin to start the job
+
+We are using configuration:
/gobblin-modules/gobblin-kafka-09/src/test/resources/kafkaHDFSStreaming.conf to
execute the job.
Review comment:
"We are using" -> "We use the"
##########
File path:
gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java
##########
@@ -217,7 +217,7 @@ public void putAll(String storeName, String tableName,
Collection<T> states) thr
}
protected void renamePath(Path tmpTablePath, Path tablePath) throws
IOException {
- HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath);
+ HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath, true);
Review comment:
We are changing the behavior of renamePath(). I am assuming you need
this to overwrite the watermarks.
##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming
manner. Normally it's running on yarn, to get the full function,
+you need have helix cluster, yarn cluster, kafka server and schema registry
set up. But we also have one local mode of streaming task using EmbeddedGobblin
+for user to easily start and test out the job. Here is the steps:
+
+# Setup local kafka cluster
+
+Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up
your kafka cluster, and create test topic "testEvents"
Review comment:
Drop "Just".
##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming
manner. Normally it's running on yarn, to get the full function,
+you need have helix cluster, yarn cluster, kafka server and schema registry
set up. But we also have one local mode of streaming task using EmbeddedGobblin
+for user to easily start and test out the job. Here is the steps:
+
+# Setup local kafka cluster
+
+Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up
your kafka cluster, and create test topic "testEvents"
+
+# Run EmbeddedGobblin to start the job
+
+We are using configuration:
/gobblin-modules/gobblin-kafka-09/src/test/resources/kafkaHDFSStreaming.conf to
execute the job.
+
+To run the job, in your intellij, you can run the test in
/gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/kafka/KafkaStreamingLocalTest
+by remove the line '(enabled=false)'. In order to run the test in IDE, you may
need to manually delete log4j-over-slf4j jar in IDE
Review comment:
remove -> removing.
##########
File path:
gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaStreamingExtractor.java
##########
@@ -323,8 +323,10 @@ public KafkaStreamingExtractor(WorkUnitState state) {
@Override
public S getSchema() {
try {
- Schema schema = (Schema)
this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName());
- return (S) schema;
+ if(this._schemaRegistry.isPresent()) {
+ return (S)(Schema)
this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName());
+ }
+ return (S) this.topicPartitions.iterator().next().getTopicName();
Review comment:
return null? Don't understand why we are returning topic name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 641238)
Time Spent: 1h 20m (was: 1h 10m)
> Create local mode of streaming kafka job to help user quickly onboard
> ---------------------------------------------------------------------
>
> Key: GOBBLIN-1521
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1521
> Project: Apache Gobblin
> Issue Type: Test
> Reporter: Zihan Li
> Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Using embedded gobblin to run streaming job, and create a guid to help user
> quickly onboard
--
This message was sent by Atlassian Jira
(v8.3.4#803005)