Hi Jaehyeon, Do you have any suggestions on how I can show more information for debugging this python harness process error? Thanks for your help.
On Mon, Jan 27, 2025 at 5:44 PM Jaehyeon Kim <dott...@gmail.com> wrote: > Your original log messages include error messages in this file - > https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L253 > . > > It may be due to failure of the Python harness process. > > On Tue, 28 Jan 2025 at 06:22, Lydian Lee <tingyenlee...@gmail.com> wrote: > >> Hi, >> >> It seems like there's also similar issue from others: >> https://github.com/apache/beam/issues/29683 >> >> For me, here's how I launch the job with the following argos: >> --runner=portableRunner >> --streaming >> --environment_type=EXTERNAL >> --environment_config=localhost:50000 >> --experiments=disable_logging_submission_environment # without this, it >> is complaining on rpc error: code = Unknown desc = ; failed to retrieve >> chunk for /tmp/staged/submission_environment_dependencies.txt >> >> For launching the harness: >> /opt/apache/beam/boot --worker_pool >> >> >> I've tested the different pipeline which all works in 2.41.0, but now >> they all failed with 2.60.0. I've also tested with 2.57.0, and that one >> seems to already have the same issue. >> >> Something worth mentioning is I also need to install >> `google-cloud-storage` when bumping the version, otherwise it will complain >> about missing the google cloud library, although I am not using gcs at all. >> ``` >> Starting worker with command ['/opt/apache/beam/boot', '--id=31-1', >> '--logging_endpoint=localhost:40649', >> '--artifact_endpoint=localhost:36751', >> '--provision_endpoint=localhost:45679', '--control_endpoint=localh 2 >> 025/01/23 23:46:14 Provision info: >> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1" >> value:{bool_value:false}} >> fields:{key:"beam:option:allow_non_restored_state:v1" >> value:{bool_value:false}} fields:{key:"beam:o 2 >> 025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint >> localhost:40649 error: EOF message follows >> 2025/01/23 23:46:28 DEBUG Received signal: terminated >> 2025/01/23 23:46:28 boot.go: error logging message over FnAPI. endpoint >> localhost:40649 error: EOF message follows >> 2025/01/23 23:46:28 ERROR Failed to import GCSFileSystem; loading of this >> filesystem will be skipped. Error details: cannot import name 'storage' >> from 'google.cloud' (unknown location) >> ``` >> >> Thanks for your help. >> >
