samcrang commented on issue #4785:
URL:
https://github.com/apache/incubator-devlake/issues/4785#issuecomment-1486998007
Hi thanks for getting back to me @klesh,
It's possible I'm doing something unusual (as well as working on stuff that
is very new)--let me know if I'm chatting nonsense.
At the moment I'm building a Docker image that copies in and installs the
dependencies for our own Python plugins. So our `Dockerfile` ends up looking
something like this:
```
FROM apache/devlake:v0.16.0-beta10
RUN apt-get update && \
apt-get update && \
apt-get install -y make build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
libncurses5-dev libncursesw5-dev xz-utils tk-dev
RUN cd /tmp && \
wget https://www.python.org/ftp/python/3.11.2/Python-3.11.2.tgz && \
tar -xzvf Python-3.11.2.tgz && \
cd Python-3.11.2/ && \
./configure --with-ensurepip=install && \
make -j4 altinstall
RUN update-alternatives --install /usr/bin/python3 python3
/usr/local/bin/python3.11 1
RUN apt-get install -y python3-venv git libmariadb-dev && \
pip install pipx && \
pipx ensurepath && \
pipx install git+https://github.com/python-poetry/poetry.git
COPY plugins /app/python/plugins
RUN cd /app/python/plugins/hopper && poetry install
...
```
Much of this is due to the fact that [recent versions of `main` contain
additional things required for
`pydevlake`](https://github.com/apache/incubator-devlake/blob/29f0c5f8baa2c0e8d2efbcf261aa36adb2b8c46a/backend/Dockerfile#L114)
whereas [the released versions
don't](https://github.com/apache/incubator-devlake/blob/v0.16.0-beta10/backend/Dockerfile#L114).
However, even after that there appears to be a bug when we trigger a run of
a Blueprint:
```
devlake-devlake-1 | Wraps: (2) Error running task 1.
devlake-devlake-1 | | Wraps: (2) subtask collectHopperReleases ended
unexpectedly
devlake-devlake-1 | | Wraps: (3) remote error response:
devlake-devlake-1 | | | Traceback (most recent call last):
devlake-devlake-1 | | | File
"/app/python/plugins/hopper/hopper/main.py", line 50, in <module>
devlake-devlake-1 | | | Hopper.start()
devlake-devlake-1 | | | File
"/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/src/incubator-devlake/backend/python/pydevlake/pydevlake/plugin.py",
line 255, in start
devlake-devlake-1 | | | fire.Fire(PluginCommands(plugin))
devlake-devlake-1 | | | File
"/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/lib/python3.11/site-packages/fire/core.py",
line 141, in Fire
devlake-devlake-1 | | | component_trace = _Fire(component, args,
parsed_flag_args, context, name)
devlake-devlake-1 | | |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devlake-devlake-1 | | | File
"/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/lib/python3.11/site-packages/fire/core.py",
line 466, in _Fire
devlake-devlake-1 | | | component, remaining_args =
_CallAndUpdateTrace(
devlake-devlake-1 | | |
^^^^^^^^^^^^^^^^^^^^
devlake-devlake-1 | | | File
"/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/lib/python3.11/site-packages/fire/core.py",
line 681, in _CallAndUpdateTrace
devlake-devlake-1 | | | component = fn(*varargs, **kwargs)
devlake-devlake-1 | | | ^^^^^^^^^^^^^^^^^^^^^^
devlake-devlake-1 | | | File
"/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/src/incubator-devlake/backend/python/pydevlake/pydevlake/ipc.py",
line 48, in wrapper
devlake-devlake-1 | | | for each in ret:
devlake-devlake-1 | | | File
"/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/src/incubator-devlake/backend/python/pydevlake/pydevlake/ipc.py",
line 63, in collect
devlake-devlake-1 | | | yield from
self._plugin.collect(self._mk_context(ctx), stream)
devlake-devlake-1 | | |
^^^^^^^^^^^^^^^^^^^^^
devlake-devlake-1 | | | File
"/root/.cache/pypoetry/virtualenvs/hopper-LibYn79P-py3.11/src/incubator-devlake/backend/python/pydevlake/pydevlake/ipc.py",
line 114, in _mk_context
devlake-devlake-1 | | | scope_dict = self._parse(data[\'scope\'])
devlake-devlake-1 | | | ~~~~^^^^^^^^^
devlake-devlake-1 | | | KeyError: \'scope\'
devlake-devlake-1 | | Wraps: (4) exit status 1
devlake-devlake-1 | | Wraps: (5) exit status 1
devlake-devlake-1 | | Error types: (1) *hintdetail.withDetail (2)
*hintdetail.withDetail (3) *hintdetail.withDetail (4) *hintdetail.withDetail
(5) *exec.ExitError
devlake-devlake-1 | Error types: (1) *hintdetail.withDetail (2)
*errors.errorString',`error_name`='Error running pipeline 1.
```
This bug doesn't appear in recent versions of `main` so it appears to have
been fixed somewhere already and therefore I've not raised a bug report.
I think the real issue I'm having here isn't that there's no `latest` Docker
image but rather that it seems like it can take a long time for things on
`main` to end up in a released artifact. I'd love to be able to take advantage
of these fixes/features that appear without having to wait until they get
cherry-picked for release.
tl;dr: releases tend to lag behind what's on `main`: is there a better way
to get these changes in production that don't involve building the Docker
images ourselves? Is it even safe to do this? Should I be waiting until things
are released officially?
Sam
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
