chuckwondo commented on issue #28558:
URL: https://github.com/apache/beam/issues/28558#issuecomment-1829705687
Thanks for the link, but given that example, I think that perhaps we're
talking about 2 different things.
The coder issue I'm having is with respect to the internal beam mechanism
used to pickle objects to pass between processes. What's happening is a failure
to pickle a function I'm using, not the data I'm dealing with.
Specifically, the error occurs while attempting to pickle the function
returned by the higher-order `drop` function. Here's the specific snippet from
the code I've shown earlier:
```python
from kerchunk.combine import drop
...
| CombineReferences(
concat_dims=["time"],
identical_dims=["lat", "lon", "channel"],
mzz_kwargs={"preprocess": drop("lst_unc_sys")},
)
```
The call to `drop("lst_unc_sys")` produces a new function, which is then
used as the value of the `"preprocess"` kwarg to `MultiZarrToZarr` (under the
covers). This ends up producing the following error:
```plain
AttributeError: Can't pickle local object 'drop.<locals>.preproc'
```
That indicates that it cannot pickle the `preproc` function that is local to
`drop`.
Given that the indicated function is not a top-level function, I created a
top-level function with the same logic as the local function, and used that as
the value of `"preprocess"` in the hopes that pickling my top-level function
would fix the issue, but that produced the same error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
