[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256646#comment-16256646
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/5013
  
Fixed on master in
0e5fb0b78cd0a3ccb144071a47579eb6c3d0570a
e9e7c3372189db7e933ff59114b9ec6245838eda

Fixed on release-1.4 in
25a28ab32609c45fb8c40f717148e32fb453d2fc
9f68212603e3601e2f7a67ff93be9b15844c14da


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256647#comment-16256647
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user NicoK closed the pull request at:

https://github.com/apache/flink/pull/5013


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253232#comment-16253232
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5013
  
@NicoK could you please close this PR?

And thanks for the work!  


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253184#comment-16253184
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5013
  
Jip, the end-to-end tests are a first step but we need more.  


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253183#comment-16253183
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5013
  
I expected that we can't test automatically this for now. From my side, +1 
to merge.

We definitely need more tests that actually run against flink-dist though.


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16253174#comment-16253174
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5013
  
Unfortunately, I don't think we can test this now. I have in mind to add 
more end-to-end tests or "system tests" that run nightly on AWS or some proper 
Hadoop and they should catch it.

What do you think, @zentol?


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251564#comment-16251564
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/5013
  
This would be difficult to test - the most reasonable test would be to put 
the Hadoop JNI into the classpath and then run the end-to-end test on that but 
we don't want to include the platform-dependant jni library, I suppose


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251544#comment-16251544
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/5013
  
Changes look reasonable to me. I supposed we can't guard this with a test?


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251486#comment-16251486
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/5013

[FLINK-7973] disable JNI bridge for relocated hadoop classes in s3-fs-*

## What is the purpose of the change

If some Hadoop's JNI library is in the classpath, it will be loaded by our 
shaded, relocated hadoop classes in the `flink-s3-fs-*` filesystems as well. 
Then, however, `NativeCodeLoader#isNativeCodeLoaded` will return `true` and 
native code libraries will be tried although our relocated namespaces have no 
JNI mapping leading to errors like `java.lang.UnsatisfiedLinkError: 
org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V`.

## Brief change log

- disable native code loading (there are more users than the shown 
`JniBasedUnixGroupsMapping`) via copies of the respective `NativeCodeLoader` 
class

## Verifying this change

This change added tests and can be verified as follows:

  - Manually verified the change by running a 3 node cluster with 1 
JobManagers and 2 TaskManagers on EMR executing the `WordCount` example with an 
S3 input source:
  ```
cp ./opt/flink-s3-fs-hadoop-1.4-SNAPSHOT.jar ./lib/
./bin/flink run -m yarn-cluster -yn 2 -ys 1 -yjm 768 -ytm 1024 
./examples/batch/WordCount.jar --input s3:///
```

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): **no**
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: **no**
  - The serializers: **no**
  - The runtime per-record code paths (performance sensitive): **yes** -- 
actually, the shaded and relocated Hadoop classes may not use (potentially 
faster) JNI implementations for certain functions; depending on their use, this 
may be per record but since this only applies to the S3 filesystem access, 
performance penalties should be hidden by its access times anyway
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
  - The S3 file system connector: **yes**

## Documentation

  - Does this pull request introduce a new feature? **no**
  - If yes, how is the feature documented? **docs**


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-7973-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5013.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5013


commit 95f533d004e7373e9de03245a7984b6355209c22
Author: Nico Kruber 
Date:   2017-11-14T13:36:22Z

[FLINK-7973] disable JNI bridge for relocated hadoop classes in s3-fs-*




> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249872#comment-16249872
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/4961
  
Fixed on master in
0e5fb0b78cd0a3ccb144071a47579eb6c3d0570a
e9e7c3372189db7e933ff59114b9ec6245838eda

Fixed on release-1.4 in
25a28ab32609c45fb8c40f717148e32fb453d2fc
9f68212603e3601e2f7a67ff93be9b15844c14da


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249873#comment-16249873
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user NicoK closed the pull request at:

https://github.com/apache/flink/pull/4961


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249819#comment-16249819
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4961
  
Could you please close the PR if it doesn't auto-close?


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249805#comment-16249805
 ] 

Aljoscha Krettek commented on FLINK-7973:
-

Fixed on master in
0e5fb0b78cd0a3ccb144071a47579eb6c3d0570a
e9e7c3372189db7e933ff59114b9ec6245838eda

Fixed on release-1.4 in
25a28ab32609c45fb8c40f717148e32fb453d2fc
9f68212603e3601e2f7a67ff93be9b15844c14da


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249495#comment-16249495
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4961
  
This look even better now! Waiting for Travis and then I'll merge.

Thanks!  



> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247727#comment-16247727
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/4961
  
I did include your end-to-end tests (with some fixes) and the fixes for the 
errors they found. Should be fine now, let's see what travis says...


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244431#comment-16244431
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/4961
  
unfortunately, it seems we cannot minimize the shaded `jar` or it will not 
contain non-imported/dynamically-loaded classes like this one


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244296#comment-16244296
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4961
  
I added two end-to-end tests and it seems for presto this currently fails 
with:
```

 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The program 
execution failed: Failed to submit job ae34d825b54b7afe2e973627d396e722 
(WordCount Example)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:492)
at 
org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:444)
at 
org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
at 
org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:525)
at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:417)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:396)
at 
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:802)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:282)
at 
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1054)
at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java:1101)
at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java:1098)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1098)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Failed to 
submit job ae34d825b54b7afe2e973627d396e722 (WordCount Example)
at 
org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1325)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:447)
at 
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at 
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
at 
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at 
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at 
org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:122)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.JobException: Creating the input splits 
caused an error: null
at 
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:262)
at 
org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:801)
at 

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243725#comment-16243725
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/4961
  
looks like I created a new word: `fute` ;) - fixed the typos both of you 
found


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243722#comment-16243722
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4961
  
I think this looks good now that the comments are addressed. @zentol what 
do you think?


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243717#comment-16243717
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149636051
  
--- Diff: flink-filesystems/flink-s3-fs-presto/README.md ---
@@ -0,0 +1,28 @@
+This project is a wrapper around the S3 file system from the Presto 
project which shades all dependencies.
+Initial simple tests seem to indicate that it responds slightly faster
+and in a bit more lightweight manner to write/read/list requests, compared
+to the Hadoop s3a FS, but it has some semantic differences.
+
+We also relocate the shaded Hadoop version to allow running in a different
+setup. For this to work, however, we needed to adapt Hadoop's 
`Configuration`
+class to load a (shaded) `core-default-shaded.xml` configuration with the
+relocated class names of classes loaded via reflection
+(in the fute, we may need to extend this to `mapred-default.xml` and 
`hdfs-defaults.xml` and their respective configuration classes).
--- End diff --

nit: "fute"


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243718#comment-16243718
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149636089
  
--- Diff: flink-filesystems/flink-s3-fs-presto/README.md ---
@@ -0,0 +1,28 @@
+This project is a wrapper around the S3 file system from the Presto 
project which shades all dependencies.
+Initial simple tests seem to indicate that it responds slightly faster
+and in a bit more lightweight manner to write/read/list requests, compared
+to the Hadoop s3a FS, but it has some semantic differences.
+
+We also relocate the shaded Hadoop version to allow running in a different
+setup. For this to work, however, we needed to adapt Hadoop's 
`Configuration`
+class to load a (shaded) `core-default-shaded.xml` configuration with the
+relocated class names of classes loaded via reflection
+(in the fute, we may need to extend this to `mapred-default.xml` and 
`hdfs-defaults.xml` and their respective configuration classes).
+
+# Changing the Hadoop Version
+
+If you want to change the Hadoop version this project depends on, the 
following
+steps are required to keep the shading correct:
+
+1. copy `org/apache/hadoop/conf/Configuration.java` from the respective 
Hadoop jar file (from `com.facebook.presto.hadoop/hadoop-apache2`) to this 
project
+  - adapt the `Configuration` class by replacing `core-default.xml` with 
`core-default-shaded.xml`.
+2. copy `core-default.xml` from the respective Hadoop jar (from 
`com.facebook.presto.hadoop/hadoop-apache2`) file to this project as
+  - `src/main/resources/core-default-shaded.xml` (replacing every 
occurence of `org.apache.hadoop` with 
`org.apache.flink.fs.s3presto.shaded.org.apache.hadoop`)
+  - `src/test/resources/core-site.xml` (as is)
+3. verify the shaded jar:
+  - does not contain any unshaded classes except for 
`org.apache.flink.fs.s3presto.S3FileSystemFactory`
+  - every other classes should be under 
`org.apache.flink.fs.s3presto.shaded`
--- End diff --

nit: "classes"


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243711#comment-16243711
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149635850
  
--- Diff: flink-filesystems/flink-s3-fs-hadoop/README.md ---
@@ -0,0 +1,27 @@
+This project is a wrapper around Hadoop's s3a file system. By pulling a 
smaller dependency tree and
+shading all dependencies away, this keeps the appearance of Flink being 
Hadoop-free,
+from a dependency perspective.
+
+We also relocate the shaded Hadoop version to allow running in a different
+setup. For this to work, however, we needed to adapt Hadoop's 
`Configuration`
+class to load a (shaded) `core-default-shaded.xml` configuration with the
+relocated class names of classes loaded via reflection
+(in the fute, we may need to extend this to `mapred-default.xml` and 
`hdfs-defaults.xml` and their respective configuration classes).
+
+# Changing the Hadoop Version
+
+If you want to change the Hadoop version this project depends on, the 
following
+steps are required to keep the shading correct:
+
+1. copy `org/apache/hadoop/conf/Configuration.java` from the respective 
Hadoop jar file to this project
+  - adapt the `Configuration` class by replacing `core-default.xml` with 
`core-default-shaded.xml`.
+2. copy `core-default.xml` from the respective Hadoop jar file to this 
project as
+  - `src/main/resources/core-default-shaded.xml` (replacing every 
occurence of `org.apache.hadoop` with 
`org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop`)
+  - `src/test/resources/core-site.xml` (as is)
+3. verify the shaded jar:
+  - does not contain any unshaded classes except for 
`org.apache.flink.fs.s3hadoop.S3FileSystemFactory`
+  - every other classes should be under 
`org.apache.flink.fs.s3hadoop.shaded`
--- End diff --

nit: "classes"


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243705#comment-16243705
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149635214
  
--- Diff: flink-filesystems/flink-s3-fs-presto/README.md ---
@@ -0,0 +1,28 @@
+This project is a wrapper around the S3 file system from the Presto 
project which shades all dependencies.
+Initial simple tests seem to indicate that it responds slightly faster
+and in a bit more lightweight manner to write/read/list requests, compared
+to the Hadoop s3a FS, but it has some semantic differences.
+
+We also relocate the shaded Hadoop version to allow running in a different
+setup. For this to work, however, we needed to adapt Hadoop's 
`Configuration`
+class to load a (shaded) `core-default-shaded.xml` configuration with the
+relocated class names of classes loaded via reflection
+(in the fute, we may need to extend this to `mapred-default.xml` and 
`hdfs-defaults.xml` and their respective configuration classes).
+
+# Changing the Hadoop Version
+
+If you want to change the Hadoop version this project depends on, the 
following
+steps are required to keep the shading correct:
+
+1. copy `org/apache/hadoop/conf/Configuration.java` from the respective 
Hadoop jar file (from `com.facebook.presto.hadoop/hadoop-apache2`) to this 
project
+  - adapt the `Configuration` class by replacing `core-default.xml` with 
`core-default-shaded.xml`.
+2. copy `core-default.xml` from the respective Hadoop jar (from 
`com.facebook.presto.hadoop/hadoop-apache2`) file to this project as
+  - `src/main/resources/core-default-shaded.xml` (replacing every 
occurence of `org.apache.hadoop` with 
`org.apache.flink.fs.s3presto.shaded.org.apache.hadoop`)
+  - `src/test/resources/core-site.xml` (as is)
+3. verify the shaded jar:
+  - does not contain any unshaded classes except for 
`org.apache.flink.fs.s3presto.S3FileSystemFactory`
+  - every other classes should be under 
`org.apache.flink.fs.s3presto.shaded`
--- End diff --

"every other class" or "all other classes"


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243704#comment-16243704
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149635319
  
--- Diff: flink-filesystems/flink-s3-fs-hadoop/README.md ---
@@ -0,0 +1,27 @@
+This project is a wrapper around Hadoop's s3a file system. By pulling a 
smaller dependency tree and
+shading all dependencies away, this keeps the appearance of Flink being 
Hadoop-free,
+from a dependency perspective.
+
+We also relocate the shaded Hadoop version to allow running in a different
+setup. For this to work, however, we needed to adapt Hadoop's 
`Configuration`
+class to load a (shaded) `core-default-shaded.xml` configuration with the
+relocated class names of classes loaded via reflection
+(in the fute, we may need to extend this to `mapred-default.xml` and 
`hdfs-defaults.xml` and their respective configuration classes).
+
+# Changing the Hadoop Version
+
+If you want to change the Hadoop version this project depends on, the 
following
+steps are required to keep the shading correct:
+
+1. copy `org/apache/hadoop/conf/Configuration.java` from the respective 
Hadoop jar file to this project
+  - adapt the `Configuration` class by replacing `core-default.xml` with 
`core-default-shaded.xml`.
+2. copy `core-default.xml` from the respective Hadoop jar file to this 
project as
+  - `src/main/resources/core-default-shaded.xml` (replacing every 
occurence of `org.apache.hadoop` with 
`org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop`)
+  - `src/test/resources/core-site.xml` (as is)
+3. verify the shaded jar:
+  - does not contain any unshaded classes except for 
`org.apache.flink.fs.s3hadoop.S3FileSystemFactory`
+  - every other classes should be under 
`org.apache.flink.fs.s3hadoop.shaded`
--- End diff --

"every other class" or "all other classes"


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243703#comment-16243703
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149635569
  
--- Diff: flink-filesystems/flink-s3-fs-hadoop/README.md ---
@@ -0,0 +1,27 @@
+This project is a wrapper around Hadoop's s3a file system. By pulling a 
smaller dependency tree and
+shading all dependencies away, this keeps the appearance of Flink being 
Hadoop-free,
+from a dependency perspective.
+
+We also relocate the shaded Hadoop version to allow running in a different
+setup. For this to work, however, we needed to adapt Hadoop's 
`Configuration`
+class to load a (shaded) `core-default-shaded.xml` configuration with the
+relocated class names of classes loaded via reflection
+(in the fute, we may need to extend this to `mapred-default.xml` and 
`hdfs-defaults.xml` and their respective configuration classes).
--- End diff --

nit: "fute"


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242502#comment-16242502
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user NicoK commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149449571
  
--- Diff: flink-filesystems/flink-s3-fs-hadoop/pom.xml ---
@@ -33,6 +33,7 @@ under the License.
jar
 

+   
--- End diff --

done (also the other suggestions)


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241884#comment-16241884
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149344188
  
--- Diff: flink-filesystems/flink-s3-fs-presto/pom.xml ---
@@ -284,6 +322,7 @@ under the License.

META-INF/maven/org.apache.h*/**

META-INF/maven/org.apache.flink/flink-hadoop-fs/**

META-INF/maven/org.apache.flink/force-shading/**
+   
core-default.xml
--- End diff --

same as in the other pom


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241885#comment-16241885
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149344386
  
--- Diff: flink-filesystems/flink-s3-fs-hadoop/pom.xml ---
@@ -33,6 +33,7 @@ under the License.
jar
 

+   
--- End diff --

should we add a similar comment to the presto hadoop dependency?


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241886#comment-16241886
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4961#discussion_r149344106
  
--- Diff: flink-filesystems/flink-s3-fs-hadoop/pom.xml ---
@@ -277,6 +336,7 @@ under the License.

META-INF/maven/org.apache.commons/**

META-INF/maven/org.apache.flink/flink-hadoop-fs/**

META-INF/maven/org.apache.flink/force-shading/**
+   
core-default.xml
--- End diff --

add comment that we're using our own shaded core-default.xml


> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240742#comment-16240742
 ] 

ASF GitHub Bot commented on FLINK-7973:
---

GitHub user NicoK opened a pull request:

https://github.com/apache/flink/pull/4961

[FLINK-7973] fix shading and relocating Hhadoop for the S3 filesystems

## What is the purpose of the change

The current shading of the `flink-s3-fs-hadoop` and `flink-s3-fs-presto` 
projects also relocates Flink core classes and even some from the JDK itself. 
Additionally, the relocation of Hadoop does not work as expected since Hadoop 
loads classes based on class names in its `core-default.xml` which are unshaded 
and thus use the original namespace.

## Brief change log

- adapt the `pom.xml` of both `flink-s3-fs-hadoop` and `flink-s3-fs-presto`:
  - do not shade everything and instead define include patterns explicitly
  - only shade and relocate Flink classes imported from flink-hadoop-fs
- hack around Hadoop loading (unshaded/non-relocated) classes based on 
names in the `core-default.xml` by overwriting the `Configuration` class (we 
may need to also extend this for the `mapred-default.xml` and 
`hdfs-defaults.xml` and their respective configuration classes in the future):
  - provide a `core-default-shaded.xml` file with shaded class names and
  - copy and adapt the `Configuration` class of the respective Hadoop 
version to load this file instead of `core-default.xml`

## Verifying this change

This change can (and was) manually tested as follows:
- verify the shaded `jar` file does not contain non-relocated classes
- verify the changed `Configuration` classes reside in the shaded namespace 
where the original Hadoop `Configuration` classes would go into, e.g. 
`org.apache.flink.fs.s3hadoop.shaded.org.hadoop.conf` (look for 
`core-default-shaded.xml` string in the `Configuration.class` file)
- verify the `META-INF/services` files are still correct (name + content)

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (yes)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NicoK/flink flink-7973

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4961.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4961






> Fix service shading relocation for S3 file systems
> --
>
> Key: FLINK-7973
> URL: https://issues.apache.org/jira/browse/FLINK-7973
> Project: Flink
>  Issue Type: Bug
>Reporter: Stephan Ewen
>Assignee: Nico Kruber
>Priority: Blocker
> Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying 
> relocation patterns multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)