Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
@casidiablo @tzulitai sorry, I had a long day yesterday and misunderstood
what you mean.
And thank you @casidiablo @xiatao123 both for confirming it works.
---
Github user xiatao123 commented on the issue:
https://github.com/apache/flink/pull/4150
Got the issue fixed after apply this patch in
Release label:emr-5.11.0
Hadoop distribution:Amazon 2.7.3
Applications:Flink 1.3.2
---
Github user xiatao123 commented on the issue:
https://github.com/apache/flink/pull/4150
I ran into a similar issue in EMR. Any suggestion on how to fix it?
`Release label:emr-5.11.0
Hadoop distribution:Amazon 2.7.3
Applications:Flink 1.3.2`
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
On second thought, the change is actually a general improvement to the
packaging of the connector, so it would make sense to merge this to `master`
and `release-1.4` also.
---
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
Merging to `release-1.3` ..
---
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
@casidiablo thanks for the info!
I'll merge this for `release-1.3` then, and will keep an extra eye on
whether the problem still occurs for 1.4 / 1.5.
---
Github user casidiablo commented on the issue:
https://github.com/apache/flink/pull/4150
Since EMR only supports Flink 1.3 I had to checkout release-1.3 and compile
the connector from there. Then I was getting this `Socket not created by this
factory` error.
I then patched
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
@casidiablo did you mean that without applying this PR's patch, the current
master worked for you? Or you had to apply this patch in order for it to work?
---
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
Thanks for confirming. I'll close this ticket and PR
---
Github user casidiablo commented on the issue:
https://github.com/apache/flink/pull/4150
This actually worked for me. I ran it on EMR 5.10, compiling the kinesis
connector from the release-1.3 branch (since EMR provides Flink 1.3)
---
Github user burkaygur commented on the issue:
https://github.com/apache/flink/pull/4150
similar to @bowenli86 i am receiving the same socket error. Tried the
proposed fix in the PR, but still getting the error. Bumped the JDK version,
didnt help either. Anyone has a working
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
Yes, we do shade the `com.amazonaws.*` packages. It was fixed before to
address problems with running the Kinesis connector on older EMR versions. So I
guess that my assumption of that there are
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4150
@tzulitai Do we shade the `aws-sdk-java` in the Kinesis connector? We
should probably, and shade it in Hadoop as well. If not, this could be a cause
of the conflict...
---
If your project is
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
From the looks of https://github.com/druid-io/druid/issues/4456, could it
be that we need to update our AWS Java SDK version?
---
If your project is set up for it, you can reply to this email and
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
@bowenli86 but isn't the shading of httpcomponents in the Kinesis consumer
supposed to avoid conflicts with whatever version you're using for
S3AFileSystem?
---
If your project is set up for it,
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
My Flink job checkpoints to S3, I'm configuring S3AFileSystem shown in
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#flink-for-hadoop-27.
I doubt if this error
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4150
Hmmm, tricky one. Do parts of the code use different configuration (one
instantiates non SSL sockets, the other one assumes SSL sockets)?
---
If your project is set up for it, you can reply to
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
During testing, I saw the following exception:
```java
java.lang.IllegalStateException: Socket not created by this factory
at
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
Thanks! Just for double check: have you on your side already verified that
this works when you're using the connector with S3 (which caused the issue for
you before)?
---
If your project is set
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
ok, I kind of figured out what `promoteTransitiveDependencies` means.
I removed explicit dependencies of httpclient and httpcore from `pom.xml`,
and made sure http-components are shaded
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
I added http components to dependencies because Hadoop-aws 2.7 depends on
http components 4.2.
```
$ jar -tf flink-connector-kinesis_2.11-1.4-SNAPSHOT.jar | grep apache/http
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4150
@bowenli86 To you have to actually explicitly add the http component
dependencies to the pom.xml file, or does the "promoteTransitiveDependencies"
config in the shading already cover that?
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4150
@bowenli86 I think Kinesis is quite popular. The conflict only arises in
combination with certain Hadoop versions, that may be the cause why not
everybody experiences it.
---
If your project
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
@tzulitai @tedyu
Hi guys, I'm also wondering how much percentage of Flink users are using
Flink with Kinesis? Kinesis has so many limitations that we are considering to
move from Kinesis
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
@tzulitai I shaded httpcomponents. Does it look good?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/4150
Thanks @bowenli86 for trying that out. Please keep my updated on this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user bowenli86 commented on the issue:
https://github.com/apache/flink/pull/4150
@tzulitai shading httpclient/core sounder like a more sound fix.
I'll see if I have time to shade them and resubmit the PR.
---
If your project is set up for it, you can reply to this
27 matches
Mail list logo
