sv2000 commented on a change in pull request #3372:
URL: https://github.com/apache/gobblin/pull/3372#discussion_r695069713



##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming 
manner. Normally it's running on yarn, to get the full function, 
+you need have helix cluster, yarn cluster, kafka server and schema registry 
set up. But we also have one local mode of streaming task using EmbeddedGobblin
+for user to easily start and test out the job. Here is the steps:
+
+# Setup local kafka cluster 
+
+Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up 
your kafka cluster, and create test topic "testEvents"
+
+# Run EmbeddedGobblin to start the job
+
+We are using configuration: 
/gobblin-modules/gobblin-kafka-09/src/test/resources/kafkaHDFSStreaming.conf to 
execute the job.

Review comment:
       "We are using" -> "We use the"

##########
File path: 
gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java
##########
@@ -217,7 +217,7 @@ public void putAll(String storeName, String tableName, 
Collection<T> states) thr
   }
 
   protected void renamePath(Path tmpTablePath, Path tablePath) throws 
IOException {
-    HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath);
+    HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath, true);

Review comment:
       We are changing the behavior of renamePath(). I am assuming you need 
this to overwrite the watermarks.

##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming 
manner. Normally it's running on yarn, to get the full function, 
+you need have helix cluster, yarn cluster, kafka server and schema registry 
set up. But we also have one local mode of streaming task using EmbeddedGobblin
+for user to easily start and test out the job. Here is the steps:
+
+# Setup local kafka cluster 
+
+Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up 
your kafka cluster, and create test topic "testEvents"

Review comment:
       Drop "Just". 

##########
File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md
##########
@@ -0,0 +1,30 @@
+# Table of Contents
+
+[TOC]
+
+# Introduction
+
+Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming 
manner. Normally it's running on yarn, to get the full function, 
+you need have helix cluster, yarn cluster, kafka server and schema registry 
set up. But we also have one local mode of streaming task using EmbeddedGobblin
+for user to easily start and test out the job. Here is the steps:
+
+# Setup local kafka cluster 
+
+Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up 
your kafka cluster, and create test topic "testEvents"
+
+# Run EmbeddedGobblin to start the job
+
+We are using configuration: 
/gobblin-modules/gobblin-kafka-09/src/test/resources/kafkaHDFSStreaming.conf to 
execute the job.
+
+To run the job, in your intellij, you can run the test in 
/gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/kafka/KafkaStreamingLocalTest
+by remove the line '(enabled=false)'. In order to run the test in IDE, you may 
need to manually delete log4j-over-slf4j jar in IDE 

Review comment:
       remove -> removing. 

##########
File path: 
gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaStreamingExtractor.java
##########
@@ -323,8 +323,10 @@ public KafkaStreamingExtractor(WorkUnitState state) {
   @Override
   public S getSchema() {
     try {
-      Schema schema = (Schema) 
this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName());
-      return (S) schema;
+      if(this._schemaRegistry.isPresent()) {
+        return (S)(Schema) 
this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName());
+      }
+      return (S) this.topicPartitions.iterator().next().getTopicName();

Review comment:
       return null? Don't understand why we are returning topic name.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to