[ 
https://issues.apache.org/jira/browse/STORM-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15374100#comment-15374100
 ] 

ASF GitHub Bot commented on STORM-1910:
---------------------------------------

Github user harshach commented on a diff in the pull request:

    https://github.com/apache/storm/pull/1558#discussion_r70550095
  
    --- Diff: 
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java ---
    @@ -291,10 +361,11 @@ public void open(Map conf, TopologyContext context, 
SpoutOutputCollector collect
         this.tupleCounter = 0;
     
         // Hdfs related settings
    -    if( conf.containsKey(Configs.HDFS_URI)) {
    +    if(this.hdfsUri==null && conf.containsKey(Configs.HDFS_URI)) {
    --- End diff --
    
    can you space after if


> One topology can't use hdfs spout to read from two locations
> ------------------------------------------------------------
>
>                 Key: STORM-1910
>                 URL: https://issues.apache.org/jira/browse/STORM-1910
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-hdfs
>    Affects Versions: 1.0.1
>            Reporter: Raghav Kumar Gautam
>            Assignee: Roshan Naik
>             Fix For: 1.1.0
>
>
> The hdfs uri is passed using config:
> {code}
>     conf.put(Configs.HDFS_URI, hdfsUri);
> {code}
> I see two problems with this approach:
> 1. If someone wants to used two hdfsUri in same or different spouts - then 
> that does not seem feasible.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/examples/storm-starter/src/jvm/storm/starter/HdfsSpoutTopology.java#L117-L117
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L331-L331
> {code}
>     if ( !conf.containsKey(Configs.SOURCE_DIR) ) {
>       LOG.error(Configs.SOURCE_DIR + " setting is required");
>       throw new RuntimeException(Configs.SOURCE_DIR + " setting is required");
>     }
>     this.sourceDirPath = new Path( conf.get(Configs.SOURCE_DIR).toString() );
> {code}
> 2. It does not fail fast i.e. at the time of topology submissing. We can fail 
> fast if the hdfs path is invalid or credentials/permissions are not ok. Such 
> errors at this time can only be detected at runtime by looking at the worker 
> logs.
> https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L297-L297



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to