On 19. Dec 2017, at 20:10, shashank agarwal
<shashank...@gmail.com <mailto:shashank...@gmail.com>>
wrote:
Hi Timo,
I am using Rocksdbstatebackend with hdfs path. I have
following flink dependencies in my sbt :
"org.slf4j" % "slf4j-log4j12" % "1.7.21",
"org.apache.flink" %% "flink-scala" % flinkVersion %
"provided",
"org.apache.flink" %% "flink-streaming-scala" %
flinkVersion % "provided",
"org.apache.flink" %% "flink-cep-scala" % flinkVersion,
"org.apache.flink" %% "flink-connector-kafka-0.10" %
flinkVersion,
"org.apache.flink" %% "flink-connector-filesystem" %
flinkVersion,
"org.apache.flink" %% "flink-statebackend-rocksdb" %
flinkVersion,
"org.apache.flink" %% "flink-connector-cassandra" % "1.3.2",
"org.apache.flink" % "flink-shaded-hadoop2" % flinkVersion,
when i start flink yarn session it's working fine even
it's creating flink checkpointing directory and copying
libs into hdfs.
But when I submit the application to this yarn session
it prints following logs :
Using the result of 'hadoop classpath' to augment the
Hadoop classpath:
/usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.
<http://2.6.0.0/>3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.
<http://2.6.0.>3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//*
But application fails contuniously with logs which i
have sent earlier.
I have tried to add flink- hadoop-compability*.jar as
suggested by Jorn but it's not working.
On Tue, Dec 19, 2017 at 5:08 PM, shashank
agarwal<shashank...@gmail.com
<mailto:shashank...@gmail.com>>wrote:
yes, it's working fine. now not getting compile time
error.
But when i trying to run this on cluster or yarn,
getting following runtime error :
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
Could not find a file system implementation for scheme 'hdfs'. The scheme is
not directly supported by Flink and no Hadoop file system to support this
scheme could be loaded.
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405)
at
org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at
org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:273)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop File
System abstraction does not support scheme 'hdfs'. Either no file system
implementation exists for that scheme, or the relevant classes are missing from
the classpath.
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:102)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
... 12 more
Caused by: java.io.IOException: No FileSystem for scheme: hdfs
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786)
at
org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:99)
... 13 more
while submitting job it's printing following logs so i think
it's including hdoop libs :
Using the result of 'hadoop classpath' to augment
the Hadoop classpath:
/usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.
<http://2.6.0.0/>3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.
<http://2.6.0.>3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//*
On Fri, Dec 8, 2017 at 9:24 PM, shashank
agarwal<shashank...@gmail.com
<mailto:shashank...@gmail.com>>wrote:
Sure i’ll Try that. Thanks
On Fri, 8 Dec 2017 at 9:18 PM, Stephan Ewen
<se...@apache.org <mailto:se...@apache.org>> wrote:
I would recommend to add
"flink-shaded-hadoop2". That is a bundle of
all Hadoop dependencies used by Flink.
On Fri, Dec 8, 2017 at 3:44 PM, Aljoscha
Krettek<aljos...@apache.org
<mailto:aljos...@apache.org>>wrote:
I see, thanks for letting us know!
On 8. Dec 2017, at 15:42, shashank
agarwal <shashank...@gmail.com
<mailto:shashank...@gmail.com>> wrote:
I had to include two dependencies.
hadoop-hdfs (this for HDFS configuration)
hadoop-common (this for Path)
On Fri, Dec 8, 2017 at 7:38 PM,
Aljoscha Krettek<aljos...@apache.org
<mailto:aljos...@apache.org>>wrote:
I think hadoop-hdfs might be
sufficient.
On 8. Dec 2017, at 14:48, shashank
agarwal <shashank...@gmail.com
<mailto:shashank...@gmail.com>> wrote:
Can you specifically guide which
dependencies should I add to
extend this :
https://github.com/apache/flink/blob/release-1.4.0-rc3/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/Bucketer.java
<https://github.com/apache/flink/blob/release-1.4.0-rc3/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/Bucketer.java>
is the Hadoop-core is sufficient.
On Fri, Dec 8, 2017 at 6:58 PM,
shashank
agarwal<shashank...@gmail.com
<mailto:shashank...@gmail.com>>wrote:
It's a compilation error. I
think I have to include the
Hadoop dependencies.
On Fri, Dec 8, 2017 at 6:54
PM, Aljoscha
Krettek<aljos...@apache.org
<mailto:aljos...@apache.org>>wrote:
Hi,
Is this a compilation
error or at runtime. But
in general, yes you have
to include the Hadoop
dependencies if they're
not there.
Best,
Aljoscha
On 8. Dec 2017, at 14:10,
shashank agarwal
<shashank...@gmail.com
<mailto:shashank...@gmail.com>>
wrote:
Hello,
I am trying to test
1.4.0-RC3, Hadoop
libraries removed in this
version. Actually, i have
created custom
Bucketer for the
bucketing sink. I am
extending
org.apache.flink.streaming.connectors.fs.bucketing.Bucketer
in the class, i have to
use org.apache.hadoop.fs.Path
but as hadoop libraries
removed it's giving error
"object hadoop is not a
member of package org.apache"
Should i have to include
Hadoop client libs in
build.sbt dependencies.
Thanks
Shashank
--
Thanks Regards
SHASHANK AGARWAL
--- Trying to mobilize the
things....
--
Thanks Regards
SHASHANK AGARWAL
--- Trying to mobilize the
things....
--
Thanks Regards
SHASHANK AGARWAL
--- Trying to mobilize the things....
--
Sent from iPhone 5
--
Thanks Regards
SHASHANK AGARWAL
--- Trying to mobilize the things....
--
Thanks Regards
SHASHANK AGARWAL
--- Trying to mobilize the things....
