[GitHub] [flink] xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-603067473 @tillrohrmann Thanks for the review. I've addressed the comments. Regarding queryable state, I did take a look but couldn't find an easy way to separate the external and bind address/port for it. It uses Netty `ServerBootstrap` (in `AbstractServerBase`), which seems not providing any API for separating external & bind. I guess it requires more background knowledge than I have to touch this feature. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-602522424 @tillrohrmann I'm ok with supporting the scenarios where address of `t` can be resolved by all other `t'` except for itself, and have updated the PR to address this issue.After all, it is indeed an improvement that we are able to support more use cases without introducing any problem to exist ones. It just how tricky I have to simulate this scenario in the e2e test that makes me suspect something might be wrong. Or put it another way, why users want to use Flink like this? Have we truly understood what they need? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-602404885 @flinkbot run travis re-run the last Travis build @flinkbot run azure re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-602363516 @tillrohrmann, I have addressed your comments. However, I'm still trying to understand how the following is a common need. > What we actually need is that for a given TM `t` all other TMs `t'` can resolve the address of `t`. I find it not natural to simulate the above scenario with Docker in the e2e test. For each container I have to manually add the hostname-ip entries of all other containers except for itself. Say all the JM and TMs are running in Docker containers. If a given TM `t` cannot resolve its own external address from inside the container, then unless specifically configured like I did in the e2e test, how could the JM and all the other TMs `t'`, which are also inside containers, resolve the address of `t`? If `t'` can resolve the address of `t` by using the same DNS settings as its host, then shouldn't `t'` also use the same DNS settings as its host and be able to resolve its own address? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-601638616 Thanks for the responses, @tillrohrmann. I'll try to address the comments asap, including the unresolvable hostname issue. BTW, I think you overlooked two comments regarding the binding port range, which are in the "hidden conversations". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-601536922 Thanks for the review @tillrohrmann. > A quick question for my understanding. In the e2e test we give every container an external address which is equal to the hosts ip address, right? How can the docker container route the packages for this address if the docker network does not happen to use the same subnet as the host's IP address? Can docker container talk to services running in the host's network? I guess I don't understand Docker good enough... As I understand, all the packages whose destination is not the docker network will be routed to the host. It depends on the host's setting whether and where to further route the package, which, without any special settings, should be simply forward the package like any other package sent from the host. Accessing other services running on the host should have no difference from accessing any public Internet address from inside a docker container. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-595001635 @tillrohrmann I checked the test failures. The failures on Azure are all unrelated. The NAT e2e test failure on Travis is cause by error message in logs. - The job is executed successfully, with correct result outputted. - The error message is about the RM RPC service not started yet when one of the TMs tries to connect to it. - The following JM log shows that the message is successfully received before discarded, indicating the TM has no problem resolving the correct RM address and RPC port. `The rpc endpoint org.apache.flink.runtime.resourcemanager.StandaloneResourceManager has not been started yet. Discarding message org.apache.flink.runtime.rpc.messages.RemoteFencedMessage until processing is started.` - I think this is not a real problem, because TM will retry connecting to RM later. I've set `skip_check_exceptions` for this test case. I think relying on the result hash check should be enough. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-594996261 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT.
xintongsong commented on issue #11284: [FLINK-15911][runtime] Make Flink work with NAT. URL: https://github.com/apache/flink/pull/11284#issuecomment-593843946 Thanks @KarmaGYZ, I've addressed the comments, and travis has gave a green light. https://travis-ci.org/xintongsong/flink/builds/657623133 @tillrohrmann, @zhijiangW Could you please also help take a look at this PR? Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
