[
https://issues.apache.org/jira/browse/BEAM-8324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941140#comment-16941140
]
Valentyn Tymofieiev edited comment on BEAM-8324 at 9/30/19 5:01 PM:
--------------------------------------------------------------------
I think the cause of the error is that we pickle main session using
dill==0.3.1.1, and unpickle it using dill==0.3.0 (which is installed on DF
workers).
We can update the version of Dill in DF workers, or restrict the upperbound
for dill, which may be advisable given [~yoshiki.obata]'s report in
https://issues.apache.org/jira/browse/BEAM-5878?focusedCommentId=16941114&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16941114.
was (Author: tvalentyn):
I think the cause of the error is that we pickle main session using
dill==0.3.1.1, and unpickle it using dill==0.2.9 (which is installed on DF
workers).
We can update the version of Dill in DF workers, or restrict the upperbound
for dill, which may be advisable given [~yoshiki.obata]'s report in
https://issues.apache.org/jira/browse/BEAM-5878?focusedCommentId=16941114&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16941114.
> Pre/Postcommit Dataflow IT tests fail: _create_function() takes from 2 to 6
> positional arguments but 7 were given
> -----------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-8324
> URL: https://issues.apache.org/jira/browse/BEAM-8324
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Fix For: 2.16.0
>
>
> {noformat}
> Dataflow pipeline failed. State: FAILED, Error:
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line
> 773, in run
> self._load_main_session(self.local_staging_directory)
> File
> "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line
> 489, in _load_main_session
> pickler.load_session(session_file)
> File
> "/usr/local/lib/python3.7/site-packages/apache_beam/internal/pickler.py",
> line 287, in load_session
> return dill.load_session(file_path)
> File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 410, in
> load_session
> module = unpickler.load()
> TypeError: _create_function() takes from 2 to 6 positional arguments but 7
> were given
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
