Hello .
>From this line in the log, may I confirm whether you set up harness container ?
I have executed your code in below environment and it worked. Python
version is newer though.
OS: Fedora 31
BEAM: 2.19.0
PYTHON: 3.7.6
Compared to my log , I realized below line does not exist in your log.
--------------------------------------------------------------------------------------------
357 [grpc-default-executor-1] INFO
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService
- Beam Fn Control client connected with id 1-1
358 [grpc-default-executor-1] INFO
org.apache.beam.runners.fnexecution.data.GrpcDataService - Beam Fn
Data client connected.
--------------------------------------------------------------------------------------------
One thing I realized was in LOOPBACK mode for harness container , job
server uses hostname for when communicating with flink.
_______________________________________________________
Caused by: java.net.UnknownHostException: fedora-desktop: Name or
service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
... 34 more
_______________________________________________________
Perhaps temporarily adding your hostname to "hosts file" by resolving
with 127.0.0.1 and see if it works ?
Thanks,
Yu Watanabe
On Mon, Feb 10, 2020 at 3:29 AM Xander Song <[email protected]> wrote:
>
> Hi Jincheng,
>
> Thanks for your help. Yes, I am using Mac. Your suggestion allowed me to
> submit the job on port 8099. However, I am now encountering a different error
> message.
>
> ERROR:root:java.net.ConnectException: Connection refused
>
> Traceback (most recent call last):
>
> File "test_beam_local_flink.py", line 18, in <module>
>
> | apache_beam.Map(lambda x: print(x))
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py",
> line 481, in __exit__
>
> self.run().wait_until_finish()
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py",
> line 461, in run
>
> self._options).run(False)
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py",
> line 474, in run
>
> return self.runner.run_pipeline(self, self._options)
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 334, in run_pipeline
>
> result.wait_until_finish()
>
> File
> "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 455, in wait_until_finish
>
> self._job_id, self._state, self._last_error_message()))
>
> RuntimeError: Pipeline
> BeamApp-root-0209182021-fc6ec677_c351e1bb-d428-4475-9910-4544d8ec1de4 failed
> in state FAILED: java.net.ConnectException: Connection refused
>
>
> I've attached a text file containing the output from the terminal running the
> Docker container in case that is informative (it is quite lengthy). Any
> suggestions are appreciated.
>
> Best,
> Xander
>
> On Sun, Feb 9, 2020 at 2:44 AM jincheng sun <[email protected]> wrote:
>>
>>
>> Hi Xander,
>>
>> Are you using Mac? The option --net=host doesn't work on Mac[1]. Could you
>> try to see if the command
>> `docker run -p 8099:8099 -p 8098:8098 -p 8097:8097
>> apachebeam/flink1.9_job_server:latest` works?
>>
>> [1] https://forums.docker.com/t/should-docker-run-net-host-work/14215/28
>>
>> Best,
>> Jincheng
>> -------------
>> Twitter: https://twitter.com/sunjincheng121
>> -------------
>>
>>
>> Xander Song <[email protected]> 于2020年2月9日周日 上午11:52写道：
>>>
>>> Do you have any suggestions for addressing this issue? I am unsure of what
>>> to try next.
>>>
>>> On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka <[email protected]> wrote:
>>>>
>>>> That seems to be a problem.
>>>>
>>>> When I try the command, I get
>>>>
>>>> $ telnet localhost 8099
>>>> Trying ::1...
>>>> Connected to localhost.
>>>> Escape character is '^]'.
>>>> �^CConnection closed by foreign host.
>>>>
>>>> On Fri, Feb 7, 2020 at 5:34 PM Xander Song <[email protected]> wrote:
>>>>>
>>>>> Thanks for the response. After entering telnet localhost 8099, I receive
>>>>>
>>>>> Trying ::1...
>>>>>
>>>>> telnet: connect to address ::1: Connection refused
>>>>>
>>>>> Trying 127.0.0.1...
>>>>>
>>>>> telnet: connect to address 127.0.0.1: Connection refused
>>>>>
>>>>> telnet: Unable to connect to remote host
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka <[email protected]> wrote:
>>>>>>
>>>>>> Seems that pipeline submission from sdk is not able to reach the job
>>>>>> server which was started in docker.
>>>>>>
>>>>>> Can you try running "telnet localhost 8099" to make sure that pipeline
>>>>>> submission can reach the job server.
>>>>>>
>>>>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song <[email protected]> wrote:
>>>>>>>
>>>>>>> I am having difficulty following the Python guide for running Beam on
>>>>>>> Flink. I created a virtual environment with Apache Beam installed, then
>>>>>>> I started up the JobService Docker container with
>>>>>>>
>>>>>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>>>>>
>>>>>>>
>>>>>>> I receive the following message confirming that the container is
>>>>>>> running.
>>>>>>>
>>>>>>>
>>>>>>> [main] INFO
>>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>>> ArtifactStagingService started on localhost:8098
>>>>>>>
>>>>>>> [main] INFO
>>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>>> Java ExpansionService started on localhost:8097
>>>>>>>
>>>>>>> [main] INFO
>>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>>> JobService started on localhost:8099
>>>>>>>
>>>>>>>
>>>>>>> In another terminal, I execute a Beam script called
>>>>>>> test_beam_local_flink.py based on the example.
>>>>>>>
>>>>>>>
>>>>>>> from __future__ import print_function
>>>>>>>
>>>>>>> import apache_beam
>>>>>>> from apache_beam.options.pipeline_options import PipelineOptions
>>>>>>>
>>>>>>> data = [1,2,3]
>>>>>>>
>>>>>>> options = PipelineOptions([
>>>>>>> "--runner=PortableRunner",
>>>>>>> "--job_endpoint=localhost:8099",
>>>>>>> "--environment_type=LOOPBACK"
>>>>>>> ])
>>>>>>>
>>>>>>> with apache_beam.Pipeline(options=options) as p:
>>>>>>> video_collection = (
>>>>>>> p | apache_beam.Create(data)
>>>>>>> | apache_beam.Map(lambda x: x + 1)
>>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>>> )
>>>>>>>
>>>>>>> print('Done')
>>>>>>>
>>>>>>> After a wait, I get the following traceback.
>>>>>>>
>>>>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84:
>>>>>>> UserWarning: You are using Apache Beam with Python 2. New releases of
>>>>>>> Apache Beam will soon support Python 3 only.
>>>>>>>
>>>>>>> 'You are using Apache Beam with Python 2. '
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>>
>>>>>>> File "test_beam_local_flink.py", line 18, in <module>
>>>>>>>
>>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>>>>>> line 481, in __exit__
>>>>>>>
>>>>>>> self.run().wait_until_finish()
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>>>>>> line 461, in run
>>>>>>>
>>>>>>> self._options).run(False)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>>>>>> line 474, in run
>>>>>>>
>>>>>>> return self.runner.run_pipeline(self, self._options)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>>>>>>> line 220, in run_pipeline
>>>>>>>
>>>>>>> job_service = self.create_job_service(options)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>>>>>>> line 136, in create_job_service
>>>>>>>
>>>>>>> return server.start()
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py",
>>>>>>> line 59, in start
>>>>>>>
>>>>>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>>>>>>> line 140, in result
>>>>>>>
>>>>>>> self._block(timeout)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>>>>>>> line 86, in _block
>>>>>>>
>>>>>>> raise grpc.FutureTimeoutError()
>>>>>>>
>>>>>>> grpc.FutureTimeoutError
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any help is greatly appreciated.
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
