sv2000 commented on a change in pull request #3372: URL: https://github.com/apache/gobblin/pull/3372#discussion_r695069713
########## File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md ########## @@ -0,0 +1,30 @@ +# Table of Contents + +[TOC] + +# Introduction + +Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming manner. Normally it's running on yarn, to get the full function, +you need have helix cluster, yarn cluster, kafka server and schema registry set up. But we also have one local mode of streaming task using EmbeddedGobblin +for user to easily start and test out the job. Here is the steps: + +# Setup local kafka cluster + +Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up your kafka cluster, and create test topic "testEvents" + +# Run EmbeddedGobblin to start the job + +We are using configuration: /gobblin-modules/gobblin-kafka-09/src/test/resources/kafkaHDFSStreaming.conf to execute the job. Review comment: "We are using" -> "We use the" ########## File path: gobblin-metastore/src/main/java/org/apache/gobblin/metastore/FsStateStore.java ########## @@ -217,7 +217,7 @@ public void putAll(String storeName, String tableName, Collection<T> states) thr } protected void renamePath(Path tmpTablePath, Path tablePath) throws IOException { - HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath); + HadoopUtils.renamePath(this.fs, tmpTablePath, tablePath, true); Review comment: We are changing the behavior of renamePath(). I am assuming you need this to overwrite the watermarks. ########## File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md ########## @@ -0,0 +1,30 @@ +# Table of Contents + +[TOC] + +# Introduction + +Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming manner. Normally it's running on yarn, to get the full function, +you need have helix cluster, yarn cluster, kafka server and schema registry set up. But we also have one local mode of streaming task using EmbeddedGobblin +for user to easily start and test out the job. Here is the steps: + +# Setup local kafka cluster + +Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up your kafka cluster, and create test topic "testEvents" Review comment: Drop "Just". ########## File path: gobblin-docs/user-guide/Run-Gobblin-Streaming-kafka-hdfs-Locally.md ########## @@ -0,0 +1,30 @@ +# Table of Contents + +[TOC] + +# Introduction + +Gobblin streaming enable user to ingest data from Kafka to HDFS in streaming manner. Normally it's running on yarn, to get the full function, +you need have helix cluster, yarn cluster, kafka server and schema registry set up. But we also have one local mode of streaming task using EmbeddedGobblin +for user to easily start and test out the job. Here is the steps: + +# Setup local kafka cluster + +Just follow [kafka quick start](https://kafka.apache.org/quickstart) to set up your kafka cluster, and create test topic "testEvents" + +# Run EmbeddedGobblin to start the job + +We are using configuration: /gobblin-modules/gobblin-kafka-09/src/test/resources/kafkaHDFSStreaming.conf to execute the job. + +To run the job, in your intellij, you can run the test in /gobblin-modules/gobblin-kafka-09/src/test/java/org/apache/gobblin/kafka/KafkaStreamingLocalTest +by remove the line '(enabled=false)'. In order to run the test in IDE, you may need to manually delete log4j-over-slf4j jar in IDE Review comment: remove -> removing. ########## File path: gobblin-modules/gobblin-kafka-common/src/main/java/org/apache/gobblin/source/extractor/extract/kafka/KafkaStreamingExtractor.java ########## @@ -323,8 +323,10 @@ public KafkaStreamingExtractor(WorkUnitState state) { @Override public S getSchema() { try { - Schema schema = (Schema) this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName()); - return (S) schema; + if(this._schemaRegistry.isPresent()) { + return (S)(Schema) this._schemaRegistry.get().getLatestSchemaByTopic(this.topicPartitions.get(0).getTopicName()); + } + return (S) this.topicPartitions.iterator().next().getTopicName(); Review comment: return null? Don't understand why we are returning topic name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
