[
https://issues.apache.org/jira/browse/YETUS-15?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663938#comment-16663938
]
Allen Wittenauer edited comment on YETUS-15 at 10/25/18 4:33 PM:
-----------------------------------------------------------------
Thanks! This is a great catch but it also requires a lot more explanation of
what is going on in my head. haha.
bq. Should it be also installed in Dockerfile?
Yes, and in a later patch in my tree (part of YETUS-681), I do exactly that.
Sorry, I forgot to forward port that one. I also just realized I need to do
the bundle install for the website. I'm thinking that should probably be in a
3rd Dockerfile...
bq. I couldn't catch how Dockerfile on the top level is used. Is this for test
purpose or an example?
... because I've got two goals for how docker interacts with Yetus.
I gained a lot of big insights from working on YETUS-681. Yetus in its current
state is VERY HARD to use on systems like Circle CI and Gitlab CI that require
a docker image. In my head, this means we need to provide two images:
a) One with all of the pre-reqs to run all of Yetus
b) One with a working Yetus *embedded* in /usr
The former is almost what we have today in
precommit/test-patch-docker/Dockerfile (now
precommit/src/main/shell/test-patch-docker/Dockerfile in the new layout)...
except python-dateutil is missing. The latter is new and is what is more fully
baked in YETUS-681:
{code}
docker run -it -v /mysrc:/mysrc -v /mypatchdir:/mypatchdir apache/yetus:latest
qbt --basedir=/mysrc --patch-dir=/mypatchdir --plugins=all
{code}
works for a manual run. With YETUS-15, it works but requires significantly
more flags. In my head, this makes adoption of Yetus extremely easy for people
who don't want to roll their own images (what we ship is fine) or at least want
to spend less effort on it. For example, their own Dockerfile would start with
'FROM apache/yetus:latest' and pick up where we left off, just adding in their
additional build requirements. In the case of a project like Hadoop, this
would cut their Dockerfile build time and maintenance down significantly.
In the current state, as you've discovered, releasedocmaker doesn't work in
precommit so start-build-env.sh doesn't work either. From what I've diagrammed
above, the pip install of python-dateutil should be moved. The ruby
requirements get trickier since they aren't required for execution but
definitely for start-build-env.sh. So I'm leaning towards either adding a 3rd
Dockerfile or just embedding more into start-build-env.sh to fully flesh that
idea out.
Does this make sense or have I gone off the deep-end? :)
FWIW, I've already got YETUS-683 filed to really drive these ideas in the
documentation and as a reminder that INFRA needs to get summoned to set it up
on Docker hub.
was (Author: aw):
Thanks! This is a great catch but it also requires a lot more explanation of
what is going on in my head. haha.
bq. Should it be also installed in Dockerfile?
Yes, and in a later patch in my tree, I do exactly that. Sorry, I forgot to
forward port that one. I also just realized I need to do the bundle install
for the website. I'm thinking that should probably be in a 3rd Dockerfile...
bq. I couldn't catch how Dockerfile on the top level is used. Is this for test
purpose or an example?
... because I've got two goals for how docker interacts with Yetus.
I gained a lot of big insights from working on YETUS-681. Yetus in its current
state is VERY HARD to use on systems like Circle CI and Gitlab CI that require
a docker image. In my head, this means we need to provide two images:
a) One with all of the pre-reqs to run all of Yetus
b) One with a working Yetus *embedded* in /usr
The former is almost what we have today in
precommit/test-patch-docker/Dockerfile (now
precommit/src/main/shell/test-patch-docker/Dockerfile in the new layout)...
except python-dateutil is missing. The latter is new and is what is more fully
baked in YETUS-681:
{code}
docker run -it -v /mysrc:/mysrc -v /mypatchdir:/mypatchdir apache/yetus:latest
qbt --basedir=/mysrc --patch-dir=/mypatchdir --plugins=all
{code}
works for a manual run. With YETUS-15, it works but requires significantly
more flags. In my head, this makes adoption of Yetus extremely easy for people
who don't want to roll their own images (what we ship is fine) or at least want
to spend less effort on it. For example, their own Dockerfile would start with
'FROM apache/yetus:latest' and pick up where we left off, just adding in their
additional build requirements. In the case of a project like Hadoop, this
would cut their Dockerfile build time and maintenance down significantly.
In the current state, as you've discovered, releasedocmaker doesn't work in
precommit so start-build-env.sh doesn't work either. From what I've diagrammed
above, the pip install of python-dateutil should be moved. The ruby
requirements get trickier since they aren't required for execution but
definitely for start-build-env.sh. So I'm leaning towards either adding a 3rd
Dockerfile or just embedding more into start-build-env.sh to fully flesh that
idea out.
Does this make sense or have I gone off the deep-end? :)
FWIW, I've already got YETUS-683 filed to really drive these ideas in the
documentation and as a reminder that INFRA needs to get summoned to set it up
on Docker hub.
> build environment
> -----------------
>
> Key: YETUS-15
> URL: https://issues.apache.org/jira/browse/YETUS-15
> Project: Yetus
> Issue Type: Improvement
> Components: Test Patch
> Affects Versions: 0.1.0
> Reporter: Allen Wittenauer
> Assignee: Allen Wittenauer
> Priority: Critical
> Fix For: 0.9.0
>
> Attachments: HADOOP-12137.HADOOP-12111.00.patch,
> HADOOP-12137.HADOOP-12111.01.patch, YETUS-15.000.patch, YETUS-15.002.patch,
> YETUS-15.004.patch, YETUS-15.005.patch, YETUS-15.006.patch,
> YETUS-15.007.patch, YETUS-15.008.patch, YETUS-15.009.patch,
> YETUS-15.03.patch, YETUS-15.ea-maven-001.patch
>
>
> We need to have some way to build (esp the documentation!) and unit tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
