Hello guys, We're running Spark Applications on GKE and we're using a docker image hosted on https://gcr.io/spark-operator/spark-py. Our spark application streams data from a Kafka topic containing messages in Protobuf format. Therefore, we need to deserialize these messages from their binary Protobuf format.
Locally, everything works fine. But, when we're deploying this to our GKE environment using the said docker image. It breaks. Specifically, I'm encountering this issue: Traceback (most recent call last): --- truncated --- File "/usr/lib/python3.6/site-packages/google/protobuf/descriptor.py", line 47, in <module> from google.protobuf.pyext import _message ImportError: cannot import name '_message' Upon further research, it seems that a shared object file is required. On my local machine, I can find that file: $ pwd /usr/local/lib/python3.7/site-packages/google/protobuf/pyext ls __init__.py _message.cpython-37m-darwin.so python_pb2.py __pycache__ cpp_message.py As you can see, there's a _message.cpython-37m-darwin.so file. This kind of file wasn't present in the docker image even though in both environments I installed the protobuf through pip install: pip install protobuf Upon further investigation, I found out that locally it works fine because the protobuf was installed via the protobuf-3.10.0-cp37-cp37m-macosx_10_9_intel.whl <https://files.pythonhosted.org/packages/a5/c6/a8b6a74ab1e165f0aaa673a46f5c895af8780976880c98934ae82060356d/protobuf-3.10.0-cp37-cp37m-macosx_10_9_intel.whl> wheel file. Meanwhile, in the docker image, the protobuf-3.10.0-py2.py3-none-any.whl <https://files.pythonhosted.org/packages/ad/c2/86c65136e280607ddb2e5dda19e2953c1174f9919b557d1d154574481de4/protobuf-3.10.0-py2.py3-none-any.whl>one was used. It turns out that there isn't a wheel file supported for the platform which the docker image is running. I've done further investigation and found out that the image is running on linux_x86_64 and none of the wheel files support this platform, therefore the fallback is protobuf-3.10.0-py2.py3-none-any.whl <https://files.pythonhosted.org/packages/ad/c2/86c65136e280607ddb2e5dda19e2953c1174f9919b557d1d154574481de4/protobuf-3.10.0-py2.py3-none-any.whl> . My question to you guys is: is it possible to support "linux_x86_64" and the likes? What solution do you propose? Additional details: Protobuf ---------------- Version: 3.5.1 Language: Python OS ---------------- Linux 771fb7930de6 4.9.184-linuxkit #1 SMP Tue Jul 2 22:58:16 UTC 2019 x86_64 Linux Python ---------------- Python 3.6.8 (default, Apr 8 2019, 18:17:52) [GCC 8.3.0] on linux Cheers, Teddy -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/4fd13487-fa28-456b-ae13-65bc42b8ad8e%40googlegroups.com.