srdo commented on issue #2990: STORM-3372: Fix NPE when shutting down HdfsBolt,
fix storm-hdfs tests not running
URL: https://github.com/apache/storm/pull/2990#issuecomment-516178232
Thanks for the reviews @Ethanlm and @agresch
srdo merged pull request #2990: STORM-3372: Fix NPE when shutting down
HdfsBolt, fix storm-hdfs tests not running
URL: https://github.com/apache/storm/pull/2990
This is an automated message from the Apache Git Service
srdo commented on issue #2990: STORM-3372: Fix NPE when shutting down HdfsBolt,
fix storm-hdfs tests not running
URL: https://github.com/apache/storm/pull/2990#issuecomment-516156086
I think something in the JUnit 4 support changed, so Mockito now complains
about useless stubs where
srdo commented on issue #2990: STORM-3372: Fix NPE when shutting down HdfsBolt,
fix storm-hdfs tests not running
URL: https://github.com/apache/storm/pull/2990#issuecomment-516125503
Addressed @Ethanlm's comment, also bumped JUnit to 5.5.1 so we're not
upgrading to a milestone release
srdo commented on a change in pull request #2990: STORM-3372: Fix NPE when
shutting down HdfsBolt, fix storm-hdfs tests not running
URL: https://github.com/apache/storm/pull/2990#discussion_r308393282
##
File path:
external/storm-hdfs/src/test/java/org/apache/storm/hdfs/bolt
Ethanlm commented on a change in pull request #2990: STORM-3372: Fix NPE when
shutting down HdfsBolt, fix storm-hdfs tests not running
URL: https://github.com/apache/storm/pull/2990#discussion_r308310631
##
File path:
external/storm-hdfs/src/test/java/org/apache/storm/hdfs/bolt
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/668
---
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1773
Done. PTAL #1780
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user vesense commented on the issue:
https://github.com/apache/storm/pull/1773
@HeartSaVioR OK. I'll do it right now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1773
@vesense
Seems like master branch and 1.x branch for storm-hdfs are diverged. Could
you craft pull request for 1.x branch?
---
If your project is set up for it, you can reply to this email
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1773
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user dossett commented on the issue:
https://github.com/apache/storm/pull/1773
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
[
https://issues.apache.org/jira/browse/STORM-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Dossett resolved STORM-602.
-
Resolution: Not A Problem
> HdfsBolt dies when the hadoop node is not availa
oblem" based on my
previous comment.
> HdfsBolt dies when the hadoop node is not available
> ---
>
> Key: STORM-602
> URL: https://issues.apache.org/jira/browse/STORM-602
> Project: A
://github.com/apache/storm/pull/668#issuecomment-180552919
@redsanket Are you still working on this?
> HdfsBolt takes a lot of configuration, need good defaults
> -
>
> Key: STORM-828
>
-969 and have
duped this to that.
> Add tick tuples to HDFSBolt for time-based flushing
> ---
>
> Key: STORM-1162
> URL: https://issues.apache.org/jira/browse/STORM-1162
> Pro
ick tuples to HDFSBolt for time-based flushing
> ---
>
> Key: STORM-1162
> URL: https://issues.apache.org/jira/browse/STORM-1162
> Project: Apache Storm
> Issue Type: Improvement
>
[
https://issues.apache.org/jira/browse/STORM-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987887#comment-14987887
]
Koelli Mungee commented on STORM-1162:
--
Thanks [~doss...@gmail.com]
> Add tick tuples to HDFSB
Koelli Mungee created STORM-1162:
Summary: Add tick tuples to HDFSBolt for time-based flushing
Key: STORM-1162
URL: https://issues.apache.org/jira/browse/STORM-1162
Project: Apache Storm
[
https://issues.apache.org/jira/browse/STORM-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koelli Mungee updated STORM-1162:
-
Issue Type: Improvement (was: Bug)
> Add tick tuples to HDFSBolt for time-based flush
Add unit test for tick tuples to HiveBolt and HdfsBolt
> --
>
> Key: STORM-1091
> URL: https://issues.apache.org/jira/browse/STORM-1091
> Project: Apache Storm
> Issue Type: Test
://github.com/apache/storm/pull/784
> Add unit test for tick tuples to HiveBolt and HdfsBolt
> --
>
> Key: STORM-1091
> URL: https://issues.apache.org/jira/browse/STORM-1091
> Pro
:
https://github.com/apache/storm/pull/784#issuecomment-146677423
+1
> Add unit test for tick tuples to HiveBolt and HdfsBolt
> --
>
> Key: STORM-1091
> URL: https://issues.apache.org/jira/br
Aaron Dossett created STORM-1091:
Summary: Add unit test for tick tuples to HiveBolt and HdfsBolt
Key: STORM-1091
URL: https://issues.apache.org/jira/browse/STORM-1091
Project: Apache Storm
[
https://issues.apache.org/jira/browse/STORM-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Dossett reassigned STORM-1091:
Assignee: Aaron Dossett
> Add unit test for tick tuples to HiveBolt and HdfsB
:
https://github.com/apache/storm/pull/784#issuecomment-146001197
+1
> Add unit test for tick tuples to HiveBolt and HdfsBolt
> --
>
> Key: STORM-1091
> URL: https://issues.apache.org/jira/br
the better option.
Other thoughts?
> HdfsBolt dies when the hadoop node is not available
> ---
>
> Key: STORM-602
> URL: https://issues.apache.org/jira/browse/STORM-602
> Project: Apache Storm
>
offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
+
--- End diff --
Extra blank line.
> Hdfs
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/668#discussion_r41088401
--- Diff:
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/bolt/HdfsBolt.java ---
@@ -44,6 +49,25 @@
private transient FSDataOutputStream out;
for being so patient with us.
> HdfsBolt takes a lot of configuration, need good defaults
> -
>
> Key: STORM-828
> URL: https://issues.apache.org/jira/browse/STORM-828
> Pro
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/668#issuecomment-145261119
Sorry it took me so long to review this. For the most part things look
good. I made a few comments on the HdfBolt that I would like to see reflected
in the
cPolicy;
+import org.apache.storm.hdfs.common.rotation.MoveFileAction;
+
+public class CSVFileBolt extends HdfsBolt {
+ private static String fileExtension = ".csv";
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End d
offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
+
+public HdfsBolt() {
+ this(de
offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
+
+public HdfsBolt() {
+ this(de
offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
+
+public HdfsBolt() {
+ this(de
;
private RecordFormat format;
private long offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
;
private RecordFormat format;
private long offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
+
+public HdfsBolt() {
+ this(de
offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
+
+public HdfsBolt() {
--- End diff --
;
private RecordFormat format;
private long offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
;
private RecordFormat format;
private long offset = 0;
+private static String defaultSourceDir = "/tmp/source";
+private static String defaultDestDir = "/tmp/dest";
+private static String defaultFileExtension = ".txt";
+
lable, the HdfsBolt never
recovers.") has been resolved by STORM-969.
> HdfsBolt dies when the hadoop node is not available
> ---
>
> Key: STORM-602
> URL: https://issues.apache.org/jira/browse/STORM-
[
https://issues.apache.org/jira/browse/STORM-804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Dossett resolved STORM-804.
-
Resolution: Duplicate
This is a duplicate of STORM-969, which has been resolved.
> HdfsB
:
https://github.com/apache/storm/pull/668#issuecomment-139670362
Yes I have made the changes, I will change my pull request soon
> HdfsBolt takes a lot of configuration, need good defaults
> -
>
> Ke
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/668#issuecomment-139114673
@redsanket I am not sure why this patch needs to change all these files.
Why can't we just add default values to the variables here
/bolt/AbstractHdfsBolt.java#L53
similar to what we have in hive connector
https://github.com/apache/storm/blob/master/external/storm-hive/src/main/java/org/apache/storm/hive/common/HiveOptions.java#L33
> HdfsBolt takes a lot of configuration, need good defau
them with the specified record format delimiter. If it is a ",",
all tuples will be appended by a comma. The HdfsBolt actually does this
implementation when the execute is called upon it. The TSV or CSV are just
abstractions to get the intended values based on the record delimiter. Can y
. If it is a ",",
all tuples will be appended by a comma. The HdfsBolt actually does this
implementation when the execute is called upon it. The TSV or CSV are just
abstractions to get the intended values based on the record delimiter. Can you
please let me know what exactly has to be done with an
CSVFileBolt extends HdfsBolt {
+ private static String fileExtension = ".csv";
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
For futher clarification, I was further
cPolicy;
+import org.apache.storm.hdfs.common.rotation.MoveFileAction;
+
+public class CSVFileBolt extends HdfsBolt {
+ private static String fileExtension = ".csv";
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- E
:
https://github.com/apache/storm/pull/668#issuecomment-139031908
ok that seems to be true, I will change that. Mostly it will be down to
refactoring a piece of code, we might not have to make it more specific then.
Thanks
> HdfsBolt takes a lot of configuration, need good defau
Github user redsanket commented on the pull request:
https://github.com/apache/storm/pull/668#issuecomment-139031908
ok that seems to be true, I will change that. Mostly it will be down to
refactoring a piece of code, we might not have to make it more specific then.
Thanks
---
If
format to "," for
CSV and set a default record format for TSV. But you are right, it might depend
on whether we want to make it more specific or leave the user to set the
config. It would be helpful if you can give your opinion on how I should proceed
> HdfsBolt takes a lot of config
request:
https://github.com/apache/storm/pull/668#issuecomment-136152413
@redsanket why we need CSVBolt and TSVBolt when there is
DelimitedRecordFormat which users can configure
HdfsBolt takes a lot of configuration, need good defaults
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/668#issuecomment-136152413
@redsanket why we need CSVBolt and TSVBolt when there is
DelimitedRecordFormat which users can configure
---
If your project is set up for it, you can reply to this
over this issue and I presume
currently the spout emits tuples in the form of fields and the csv or tsv bolt
joins them with the specified record format delimiter. If it is a ,, all
tuples will be appended by a comma. The HdfsBolt actually does this
implementation when the execute is called upon
extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
For futher clarification, I was further thinking over this issue
org.apache.storm.hdfs.common.rotation.MoveFileAction;
+
+public class CSVFileBolt extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
For futher
org.apache.storm.hdfs.common.rotation.MoveFileAction;
+
+public class CSVFileBolt extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
That's true, I
extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
That's true, I should have made that observation, I could
org.apache.storm.hdfs.common.rotation.MoveFileAction;
+
+public class CSVFileBolt extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
I think it would take
org.apache.storm.hdfs.common.rotation.MoveFileAction;
+
+public class CSVFileBolt extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
I guess I could set
extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
I guess I could set the RecordDefaultDelimiter to ,. Yes, thanks
extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
I think it would take much more work than that and a clearer
GitHub user redsanket opened a pull request:
https://github.com/apache/storm/pull/668
STORM-828 HdfsBolt takes a lot of configuration, need good defaults
Removing configs from HDFSFileTopology example. I have made neccessary
config and constructor changes as per my understanding
org.apache.storm.hdfs.common.rotation.MoveFileAction;
+
+public class CSVFileBolt extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
Are there any
:
https://github.com/apache/storm/pull/668
STORM-828 HdfsBolt takes a lot of configuration, need good defaults
Removing configs from HDFSFileTopology example. I have made neccessary
config and constructor changes as per my understanding. It will be nice to know
if there is anything I
extends HdfsBolt {
+ private static String fileExtension = .csv;
+
+ public CSVFileBolt(String sourceDir, String destDir) {
+super(sourceDir, destDir, fileExtension);
+ }
+}
--- End diff --
Are there any guarantees that this bolt actually writes out comma
[
https://issues.apache.org/jira/browse/STORM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanket Reddy reassigned STORM-828:
--
Assignee: Sanket Reddy
HdfsBolt takes a lot of configuration, need good defaults
is Hadoop 2.5.0-cdh5.2.0).
Could someone verify this?
HdfsBolt dies when the hadoop node is not available
---
Key: STORM-602
URL: https://issues.apache.org/jira/browse/STORM-602
Project: Apache Storm
], especially the second
issue. It seems that updating {{hadoop-client}} and {{hadoop-hdfs}} in the
[storm-hdfs
pom.xml|https://github.com/apache/storm/blob/v0.9.3/external/storm-hdfs/pom.xml]
to {{2.5.2}} (our Hadoop is Hadoop 2.5.0-cdh5.2.0).
Could someone verify this?
HdfsBolt dies when
issue. It seems that updating {{hadoop-client}} and {{hadoop-hdfs}} in
the [storm-hdfs
pom.xml|https://github.com/apache/storm/blob/v0.9.3/external/storm-hdfs/pom.xml]
to {{2.5.2}} (our Hadoop is Hadoop 2.5.0-cdh5.2.0).
Could someone verify this?)
HdfsBolt dies when the hadoop node
], especially the second
issue. It seems that updating {{hadoop-client}} and {{hadoop-hdfs}} in the
[storm-hdfs
pom.xml|https://github.com/apache/storm/blob/v0.9.3/external/storm-hdfs/pom.xml]
to {{2.5.2}} (our Hadoop is Hadoop 2.5.0-cdh5.2.0).
Could someone verify this?
HdfsBolt dies when
is Hadoop 2.5.0-cdh5.2.0).
Could someone verify this?
HdfsBolt dies when the hadoop node is not available
---
Key: STORM-602
URL: https://issues.apache.org/jira/browse/STORM-602
Project: Apache Storm
Hi All,
Two questions:
1) what version of hadoop does flux work with? My assumption was that it
should not matter, but I am getting the following error when I use a
hdfsbolt with flux which I assume implies some version mismatch. I don't
have this issue if build a topology straightforward
cluster,HdfsBolt i
use in this topology.It work normally in the beginning, then i intercept the
message for a while(30 minites), after that, the bolt write failed all the time.
I think HdfsBolt.excute should reconnect and recreate filesystem after it
keeping catch exception for a period of time
rujia created STORM-804:
---
Summary: HdfsBolt doesn't work after a period of time which caused
by network problems
Key: STORM-804
URL: https://issues.apache.org/jira/browse/STORM-804
Project: Apache Storm
I am already using hdfsbolt successfully (without hdfs HA). So, I assume
this the client java class in already in my classpath if it comes with the
hadoop jar files that I load when I run my topology, unless there is jar
specific to the hadoop HA classes which would have
Hi All,
Has anyone used HdfsBolt with hdfs in HA mode? How would you determine
which hdfs node is the active node?
thanks
Clay
Bobby,
What do you mean by client here? In this context, do you consider hdfsbolt
a client? If yes, then which configuration you are referring to? I've seen
the following, but I am not sure if I follow.
- *dfs.client.failover.proxy.provider.[nameservice ID]* - the Java class
that HDFS
Clay,
When you are using storm-hdfs connector you need to package
core-site.xml and hdfs-site.xml form you cluster into your topology
jar . You can configure the storm-hdfs bolt to pass nameserviceID
HdfsBolt bolt = new HdfsBolt()
.withFsURL(hdfs://myNameserviceID
:
Hi All,
Has anyone used HdfsBolt with hdfs in HA mode? How would you determine
which hdfs node is the active node?
thanks
Clay
as on the gateway and
should behave the same way. You can also grab them from the name node or any
of the hadoop compute nodes.
This will work for the HdfsBolt that loads default configurations from the
classpath before overriding them with any custom configurations you set for
that bolt
Hi All,
Why HdfsBolt doesn't retry when the hadoop node is down or not accessible
and dies brings down the topology with it too? I can catch the run time
exception and keep the topology going, but was wondering why the retry is
not built into HdfsBolt.
thank you
Clay
Probably the best thing to do for this is to file a JIRA at issues.apache.org
under STORM and include what it takes to reproduce the issue. - Bobby
On Tuesday, December 23, 2014 11:42 AM, clay teahouse
clayteaho...@gmail.com wrote:
Hi All,
Why HdfsBolt doesn't retry when
clay teahouse created STORM-602:
---
Summary: HdfsBolt dies when the hadoop node is not available
Key: STORM-602
URL: https://issues.apache.org/jira/browse/STORM-602
Project: Apache Storm
Issue
86 matches
Mail list logo
