usingh83 commented on a change in pull request #14531: URL: https://github.com/apache/beam/pull/14531#discussion_r630524045
########## File path: examples/java/src/main/java/org/apache/beam/examples/complete/twitterstreamgenerator/TwitterIO.java ########## @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.examples.complete.twitterstreamgenerator; + +import java.util.ArrayList; +import java.util.List; +import org.apache.beam.sdk.transforms.Create; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.transforms.ParDo; +import org.apache.beam.sdk.values.PBegin; +import org.apache.beam.sdk.values.PCollection; + +/** + * An unbounded source for <a + * href="https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data";>twitter</a> + * stream. PTransforms for streaming live tweets from twitter. Reading from Twitter is supported by + * read() + * + * <p>Standard Twitter API can be read using a list of Twitter Config + * readStandardStream(List<TwitterConfig>) + * + * <p>We allow multiple Twitter configurations to demonstrate how multiple twitter streams can be + * combined in a single pipeline. + * + * <p>TwitterIO.readStandardStream( Arrays.asList( new TwitterConfig .Builder() .setKey("") + * .setSecret("") .setToken("") .setTokenSecret("") .setFilters(Arrays.asList("", "")) + * .setLanguage("en") .setTweetsCount(10L) .setMinutesToRun(1) .build())) + */ +public class TwitterIO { + + /** + * Initializes the stream by converting input to a Twitter connection configuration. + * + * @param twitterConfigs list of twitter config + * @return PTransform of statuses + */ + public static PTransform<PBegin, PCollection<String>> readStandardStream( + List<TwitterConfig> twitterConfigs) { Review comment: Yes tested this thoroughly, with multiple configuration. I am sure the multiple configurations will be able to scale and everything. Have checked this in my local running the pipeline in different configuration multiple times. I would change this to single config if we have a counter argument against it. I am very sure for now that this way is better. I think i can have a unit test that verifies multiple configs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
