[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear
[
https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-9441:
-
Priority: Major (was: Blocker)
> Hadoop Required Dependency List Not Clear
> -
>
> Key: FLINK-9441
> URL: https://issues.apache.org/jira/browse/FLINK-9441
> Project: Flink
> Issue Type: Bug
> Components: Build System, Configuration, Documentation
>Affects Versions: 1.5.0, 1.4.2, 1.6.0
>Reporter: Razvan
>Priority: Major
> Fix For: 1.7.0
>
>
> To be able to use Apache Flink with S3, a few libraries from Hadoop
> distribution are required to be added to lib/ folder.
> This list is partially documented in
> [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html]
> (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas
> Flink supports 2.8 also.
> How to compose the dependecy list for Hadoop 2.8?
> Is it possible to bundle the dependencies in a separate archive that users
> can download?
>
> UPDATE:
> Downloaded Apache Flink 1.4.2 for Hadoop 2.7 and it seems it was compiled
> with Hadoop 2.8. I get the error here:
>
> {{"java.lang.NumberFormatException: For input string: "100M" at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:589) at
> java.lang.Long.parseLong(Long.java:631) at
> org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1319) at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:248) at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) }}
> {{..."}}
> [https://stackoverflow.com/questions/48149929/hive-1-2-metastore-service-doesnt-start-after-configuring-it-to-s3-storage-inst?rq=1]
>
> So I cannot start the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear
[
https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-9441:
Fix Version/s: 1.7.0
> Hadoop Required Dependency List Not Clear
> -
>
> Key: FLINK-9441
> URL: https://issues.apache.org/jira/browse/FLINK-9441
> Project: Flink
> Issue Type: Bug
> Components: Build System, Configuration, Documentation
>Affects Versions: 1.5.0, 1.4.2, 1.6.0
>Reporter: Razvan
>Priority: Blocker
> Fix For: 1.7.0
>
>
> To be able to use Apache Flink with S3, a few libraries from Hadoop
> distribution are required to be added to lib/ folder.
> This list is partially documented in
> [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html]
> (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas
> Flink supports 2.8 also.
> How to compose the dependecy list for Hadoop 2.8?
> Is it possible to bundle the dependencies in a separate archive that users
> can download?
>
> UPDATE:
> Downloaded Apache Flink 1.4.2 for Hadoop 2.7 and it seems it was compiled
> with Hadoop 2.8. I get the error here:
>
> {{"java.lang.NumberFormatException: For input string: "100M" at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:589) at
> java.lang.Long.parseLong(Long.java:631) at
> org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1319) at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:248) at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) }}
> {{..."}}
> [https://stackoverflow.com/questions/48149929/hive-1-2-metastore-service-doesnt-start-after-configuring-it-to-s3-storage-inst?rq=1]
>
> So I cannot start the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear
[
https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Razvan updated FLINK-9441:
--
Description:
To be able to use Apache Flink with S3, a few libraries from Hadoop
distribution are required to be added to lib/ folder.
This list is partially documented in
[https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html]
(Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas Flink
supports 2.8 also.
How to compose the dependecy list for Hadoop 2.8?
Is it possible to bundle the dependencies in a separate archive that users can
download?
UPDATE:
Downloaded Apache Flink 1.4.2 for Hadoop 2.7 and it seems it was compiled with
Hadoop 2.8. I get the error here:
{{"java.lang.NumberFormatException: For input string: "100M" at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589) at
java.lang.Long.parseLong(Long.java:631) at
org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1319) at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:248) at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) }}
{{..."}}
[https://stackoverflow.com/questions/48149929/hive-1-2-metastore-service-doesnt-start-after-configuring-it-to-s3-storage-inst?rq=1]
So I cannot start the cluster.
was:
To be able to use Apache Flink with S3, a few libraries from Hadoop
distribution are required to be added to lib/ folder.
This list is partially documented in
[https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html]
(Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas Flink
supports 2.8 also.
How to compose the dependecy list for Hadoop 2.8?
Is it possible to bundle the dependencies in a separate archive that users can
download?
UPDATE:
> Hadoop Required Dependency List Not Clear
> -
>
> Key: FLINK-9441
> URL: https://issues.apache.org/jira/browse/FLINK-9441
> Project: Flink
> Issue Type: Bug
> Components: Build System, Configuration, Documentation
>Affects Versions: 1.5.0, 1.4.2, 1.6.0
>Reporter: Razvan
>Priority: Blocker
>
> To be able to use Apache Flink with S3, a few libraries from Hadoop
> distribution are required to be added to lib/ folder.
> This list is partially documented in
> [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html]
> (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas
> Flink supports 2.8 also.
> How to compose the dependecy list for Hadoop 2.8?
> Is it possible to bundle the dependencies in a separate archive that users
> can download?
>
> UPDATE:
> Downloaded Apache Flink 1.4.2 for Hadoop 2.7 and it seems it was compiled
> with Hadoop 2.8. I get the error here:
>
> {{"java.lang.NumberFormatException: For input string: "100M" at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:589) at
> java.lang.Long.parseLong(Long.java:631) at
> org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1319) at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:248) at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) }}
> {{..."}}
> [https://stackoverflow.com/questions/48149929/hive-1-2-metastore-service-doesnt-start-after-configuring-it-to-s3-storage-inst?rq=1]
>
> So I cannot start the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear
[ https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Razvan updated FLINK-9441: -- Description: To be able to use Apache Flink with S3, a few libraries from Hadoop distribution are required to be added to lib/ folder. This list is partially documented in [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas Flink supports 2.8 also. How to compose the dependecy list for Hadoop 2.8? Is it possible to bundle the dependencies in a separate archive that users can download? UPDATE: was: To be able to use Apache Flink with S3, a few libraries from Hadoop distribution are required to be added to lib/ folder. This list is partially documented in [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas Flink supports 2.8 also. So i would like to know how to compose the dependecy list for Hadoop 2.8 and update the documentation. Also would it be possible to bundle the dependencies in a separate archive that users can download? > Hadoop Required Dependency List Not Clear > - > > Key: FLINK-9441 > URL: https://issues.apache.org/jira/browse/FLINK-9441 > Project: Flink > Issue Type: Bug > Components: Build System, Configuration, Documentation >Affects Versions: 1.5.0, 1.4.2, 1.6.0 >Reporter: Razvan >Priority: Blocker > > To be able to use Apache Flink with S3, a few libraries from Hadoop > distribution are required to be added to lib/ folder. > This list is partially documented in > [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] > (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas > Flink supports 2.8 also. > How to compose the dependecy list for Hadoop 2.8? > Is it possible to bundle the dependencies in a separate archive that users > can download? > > UPDATE: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear
[ https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Razvan updated FLINK-9441: -- Component/s: Build System > Hadoop Required Dependency List Not Clear > - > > Key: FLINK-9441 > URL: https://issues.apache.org/jira/browse/FLINK-9441 > Project: Flink > Issue Type: Bug > Components: Build System, Configuration, Documentation >Affects Versions: 1.5.0, 1.4.2, 1.6.0 >Reporter: Razvan >Priority: Blocker > > To be able to use Apache Flink with S3, a few libraries from Hadoop > distribution are required to be added to lib/ folder. > This list is partially documented in > [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] > (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas > Flink supports 2.8 also. > So i would like to know how to compose the dependecy list for Hadoop 2.8 and > update the documentation. > Also would it be possible to bundle the dependencies in a separate archive > that users can download? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear
[ https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Razvan updated FLINK-9441: -- Component/s: Configuration > Hadoop Required Dependency List Not Clear > - > > Key: FLINK-9441 > URL: https://issues.apache.org/jira/browse/FLINK-9441 > Project: Flink > Issue Type: Bug > Components: Build System, Configuration, Documentation >Affects Versions: 1.5.0, 1.4.2, 1.6.0 >Reporter: Razvan >Priority: Blocker > > To be able to use Apache Flink with S3, a few libraries from Hadoop > distribution are required to be added to lib/ folder. > This list is partially documented in > [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] > (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas > Flink supports 2.8 also. > So i would like to know how to compose the dependecy list for Hadoop 2.8 and > update the documentation. > Also would it be possible to bundle the dependencies in a separate archive > that users can download? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear
[ https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Razvan updated FLINK-9441: -- Affects Version/s: 1.6.0 > Hadoop Required Dependency List Not Clear > - > > Key: FLINK-9441 > URL: https://issues.apache.org/jira/browse/FLINK-9441 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.5.0, 1.4.2, 1.6.0 >Reporter: Razvan >Priority: Blocker > > To be able to use Apache Flink with S3, a few libraries from Hadoop > distribution are required to be added to lib/ folder. > This list is partially documented in > [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] > (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas > Flink supports 2.8 also. > So i would like to know how to compose the dependecy list for Hadoop 2.8 and > update the documentation. > Also would it be possible to bundle the dependencies in a separate archive > that users can download? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear
[ https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Razvan updated FLINK-9441: -- Description: To be able to use Apache Flink with S3, a few libraries from Hadoop distribution are required to be added to lib/ folder. This list is partially documented in [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas Flink supports 2.8 also. So i would like to know how to compose the dependecy list for Hadoop 2.8 and update the documentation. Also would it be possible to bundle the dependencies in a separate archive that users can download? was: To be able to use Apache Flink with S3, a few libraries from Hadoop distribution are required to be added to lib/ folder of the distribution. This list is partially documented in [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas Flink supports 2.8 also. So i would like to know how to compose the dependecy list for Hadoop 2.8 and update the documentation. Also would it be possible to bundle the dependencies in a separate archive that users can download? > Hadoop Required Dependency List Not Clear > - > > Key: FLINK-9441 > URL: https://issues.apache.org/jira/browse/FLINK-9441 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.5.0, 1.4.2 >Reporter: Razvan >Priority: Blocker > > To be able to use Apache Flink with S3, a few libraries from Hadoop > distribution are required to be added to lib/ folder. > This list is partially documented in > [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] > (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas > Flink supports 2.8 also. > So i would like to know how to compose the dependecy list for Hadoop 2.8 and > update the documentation. > Also would it be possible to bundle the dependencies in a separate archive > that users can download? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9441) Hadoop Required Dependency List Not Clear
[ https://issues.apache.org/jira/browse/FLINK-9441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Razvan updated FLINK-9441: -- Description: To be able to use Apache Flink with S3, a few libraries from Hadoop distribution are required to be added to lib/ folder of the distribution. This list is partially documented in [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas Flink supports 2.8 also. So i would like to know how to compose the dependecy list for Hadoop 2.8 and update the documentation. Also would it be possible to bundle the dependencies in a separate archive that users can download? was: To be able to use Apache Flink with S3, a few libraries from Hadoop are required to be added to lib/ folder of the distribution. This list is partially documented in [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas Flink supports 2.8 also. So i would like to know how to compose the dependecy list for Hadoop 2.8 and update the documentation. Also would it be possible to bundle the dependencies in a separate archive that users can download? > Hadoop Required Dependency List Not Clear > - > > Key: FLINK-9441 > URL: https://issues.apache.org/jira/browse/FLINK-9441 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.5.0, 1.4.2 >Reporter: Razvan >Priority: Blocker > > To be able to use Apache Flink with S3, a few libraries from Hadoop > distribution are required to be added to lib/ folder of the distribution. > This list is partially documented in > [https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/aws.html] > (Provide S3 FileSystem Dependency). But it refers to Hadoop 2.7 whereas > Flink supports 2.8 also. > So i would like to know how to compose the dependecy list for Hadoop 2.8 and > update the documentation. > Also would it be possible to bundle the dependencies in a separate archive > that users can download? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
