Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Tests seem to pass on my Travis. Running some more tests to ensure
everything is running smoothly. Thank you for bearing with me!
---
If your project is set up for it, you can reply to this email and
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm I have addressed some of the review feedback and rebased to upstream
master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
>
@vijikarthi Thanks for the update. Great to see the tests are passing now.
I'm curious, why did this issue only appear on Travis and not locally?
Kafka/ZK connection is unusually
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
@vijikarthi Thanks for the update. Great to see the tests are passing now.
I'm curious, why did this issue only appear on Travis and not locally?
---
If your project is set up for it, you can reply to
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm The issue is apparently due to ZK client API implementation which
lookup the system configuration property to determined the type of event
(SASLAuthenticated/SysConnected) that it should
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@nielsbasjes Thanks for the link. The issue however is related to ZK SASL
client API implementation and it took a while to figure out the actual cause.
---
If your project is set up for it, you
Github user nielsbasjes commented on the issue:
https://github.com/apache/flink/pull/2275
Perhaps you are running into similar problems as described here:
http://stackoverflow.com/questions/2890259/running-each-junit-test-in-a-separate-jvm-in-eclipse
#JustTryingToHelp
---
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Thanks, will try that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
I would suggest to just you your own Travis account. It is free, you just
need to connect it with your account. Then push to any branch in your GitHub
account and Travis will run.
---
If your project
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
hmmm.. it's getting complicated. will try to debug the issue. Do you know
why on Travis it has to fail but not on Jenkins?
How do I simulate this on Travis for my own testing?
---
If
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Interestingly, this just fails on Travis but on Jenkins the test completes:
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
I would suspect the tests suffer from is a network configuration or port
issue when they run in parallel.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Unfortunately, that didn't fix it. The builds still consistently fails on
Travis in `Kafka09SecureRunITCase`:
https://travis-ci.org/apache/flink/builds/158627315
Under these circumstances, I
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
I have noticed the issue occasionally in master branch too but very
inconsistent. I just rebased against the latest master and ran "mvn clean
verify". I don't see any errors. I have tried couple
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
We will have to fix the issue before we can merge this PR. The timeout
doesn't occur in non-secure Kafka tests. I think we need to increase the
timeout or change the test layout. Could you look into
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
rebased again with the latest master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm I believe the ZK timeout issue occurs from
LocalFlinkMiniClusterITCase->testLocalFlinkMiniClusterWithMultipleTaskManagers
test case but it is not consistent. I ran the Kafka test case alone
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
The Kafka tests fail:
```
Running org.apache.flink.streaming.connectors.kafka.Kafka09SecureRunITCase
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
zookeeper
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Thank you. Testing again and merging if nothing fails.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm, I was able to reproduce this issue on some other machine. The issue
is that KRB5 config file which is mounted as a local resource is not visible
though we set the system property
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm, I am not seeing the error messages when I run the Yarn test. Could
you please run "secure" test case alone and share the logs?
>
mvn test integration-test
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Yarn tests still not passing:
```
2016-09-02 11:35:04,244 ERROR
org.apache.flink.yarn.YarnApplicationMasterRunner - YARN
Application Master initialization failed
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Running some last tests before merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm I have rebased the code to the latest master. Please take a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Could you please rebase to the latest master and remove any merge commits?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Thanks for testing this on an actual secure cluster.
Hmpf, only allowing to bind to privileged ports doesn't really improve
security but we have to deal with this limitation somehow. I was
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
>
YARNSessionFIFOSecuredITCase gives me the following:
17:49:58,097 INFO SecurityLogger.org.apache.hadoop.ipc.Server - Auth
successful for appattempt_1471880990715_0001_01
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
>
It seems like the privileged port issues can be circumvented by setting
conf.getBoolean("dfs.datanode.require.secure.ports", false)?
It is not supported yet
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
It seems like the privileged port issues can be circumvented by setting `
conf.getBoolean("dfs.datanode.require.secure.ports", false)`?
---
If your project is set up for it, you can reply to this email
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
> If we have to use privileged ports then we won't be able to use our CI
system. Are you sure it can only run in privileged mode? Is it not possible to
change the port binding to port >= 1024?
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
>The RollingFileSink error might be due to
https://issues.apache.org/jira/browse/HDFS-9213. The secure MiniDFS cluster
requires privileged ports to be used and we need to enable the java process to
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
> YARNSessionFIFOSecuredITCase gives me the following:
17:49:58,097 INFO SecurityLogger.org.apache.hadoop.ipc.Server - Auth
successful for appattempt_1471880990715_0001_01
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
The `RollingFileSink` error might be due to
https://issues.apache.org/jira/browse/HDFS-9213. The secure MiniDFS
cluster requires privileged ports to be used and we need to enable the java
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Seems like this is not yet in a mergeable state and needs another pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
`RollingFileSink`
```
---
T E S T S
---
Running
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
YARNSessionFIFOSecuredITCase gives me the following:
>17:49:58,097 INFO SecurityLogger.org.apache.hadoop.ipc.Server
- Auth successful for appattempt_1471880990715_0001_01
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
Hi @vijikarthi! Sorry for the late reply. I intend to merge this as soon as
possible. When I run the secure test cases from IntelliJ I get
>java.lang.NoClassDefFoundError: jdbm/helper/CachePolicy
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm - Could you please take a look and let me know if you need any further
changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Thanks Max. I have reorganized the secure test case and removed custom
JRunner implementation for Kafka. Kept single secure test case for HDFS, Kafka
and Yarn modules. Please take a look.
---
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
>HDFS and Yarn are handled through the @BeforeClass and @AfterClass style
and they do not use custom JRunner implementation. As you have suggested, I
could keep just one or two tests for each of the
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm - Could you please let me know if you are okay with the modifications
to the integration test case scenarios that I have mentioned. I am open to keep
just 3 classes for each scenarios (HDFS,
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm - I have addressed most of the feedback and pushed the changes. Please
take a look when you get a chance.
Regarding the secure test cases,
- HDFS and Yarn are handled through
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
> Yes, it is not essential to run the secure test case all the time as it
consumes more cycles. Do you have any suggestion on controlling this through
some mvn/surefire plugin configuration?
I
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@mxm - Thanks for your feedback and here is my response to some of your
comments.
- Do we need to run all the Yarn tests normally and secured? We already
have problems with our test
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
@nielsbasjes - In most deployments, the KRB5 configuration file will be
located in a well known (for e.g., /etc/krb5.conf) but in scenarios where
custom location needs to be provided, we could
Github user nielsbasjes commented on the issue:
https://github.com/apache/flink/pull/2275
Question: Does this also work when starting yarn-session.sh?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user nielsbasjes commented on the issue:
https://github.com/apache/flink/pull/2275
I noticed that the system property java.security.krb5.conf isn't set
anywhere in the code.
Shouldn't there be a config setting/property that allows the user to set
this?
---
If your
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Thanks @mxm for the review and feedback. I will respond to the comments and
incorporate any changes required.
---
If your project is set up for it, you can reply to this email and have your
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2275
I've passed through your changes. Great work! Thanks a lot for the
documentation, it was very helpful. Please don't be scared by all the comments,
most of them are minor. The most important points:
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Team, please let me know if any additional details are required to
kick-start the review process?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user vijikarthi commented on the issue:
https://github.com/apache/flink/pull/2275
Adding some more cotext to the implementation details. which is based on
the design proposal
(https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing)
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2275
One quick comment (without understanding the functionality of the code):
We are trying to minimize external dependencies, especially to Guava (which
has a lot of version conflicts). If you
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2275
Thanks for that big contribution.
Could you explain the changes you made in more detail? What new parameters
were introduced, how is the Keytab forwarded. How do components interact with
53 matches
Mail list logo
