The workaround Xavier suggested seems working for me.
I think this workaround can work for the users who don't use the docker-compose.
For the users who use the docker-compose, I think the easiest
workaround is that they first create docker volumes using an old image
and then run the 0.21.0 image.
You can do this by running these commands.
$ cd ${PREV_SRC_DIR}
$ docker-compose -f distribution/docker/docker-compose.yml create
$ cd ${0.21.0_SRC_DIR}
$ docker-compose -f distribution/docker/docker-compose.yml up
This workaround seems easy enough to me. Also, given that code freeze
for the 0.22.0 release is coming, I think this workaround could be
enough for now.
Does anyone strongly think that we should do a patch release to fix this?
On Wed, Apr 28, 2021 at 2:00 PM Jihoon Son <jihoon...@apache.org> wrote:
>
> I could reproduce it. It seems that my previous testing passed because
> of the docker volume that was created a long time ago with a proper
> ownership.
> It also seems not particular to the specific OS version because I
> could reproduce the same issue in linux as well. I used the ubuntu
> linux of the kernel version of 5.4.0-70 and docker 20.10.5.
> I'm now looking for an easy workaround. Please let me know if you know
> such a workaround.
> Otherwise, maybe we should do a patch release.
>
>
> On Wed, Apr 28, 2021 at 10:35 AM Xavier Léauté
> <xav...@confluent.io.invalid> wrote:
> >
> > I'm running 3.3.1 on Big Sur, so this might be particular to the specific
> > OS version?
> >
> > On Wed, Apr 28, 2021 at 10:33 AM Jihoon Son <jihoon...@apache.org> wrote:
> >
> > > Interesting. I also used Docker Desktop 3.3.1 on the Catalina macOS
> > > for my testing. The current docker image worked fine for me.
> > > It sounds good to me to add this as a known issue in the release
> > > notes, but I hope it to be more specific.
> > > Do you have more details about the issue such as in what conditions
> > > you could see it?
> > >
> > > On Wed, Apr 28, 2021 at 9:48 AM Xavier Léauté
> > > <xav...@confluent.io.invalid> wrote:
> > > >
> > > > Hi Frank,
> > > >
> > > > I am noticing a similar problem when running on Docker for Mac.
> > > >
> > > > This appears to be specific to macOS, and how docker initializes the
> > > volume
> > > > mount point permissions.
> > > > As a workaround you can pass a volume at the command-line, which somehow
> > > > sets the right ownership.
> > > >
> > > > We might want to add a known issue related to that in the release notes.
> > > >
> > > > On Mon, Apr 26, 2021 at 7:54 PM frank chen <frankc...@apache.org> wrote:
> > > >
> > > > > I did two tests this morning.
> > > > > 1. Docker Desktop was downgraded to 3.2.0/3.0.0, this problem still
> > > exists
> > > > > 2. Druid cluster successfully started based on 0.20.0 image on the
> > > > > same
> > > > > macbook.
> > > > >
> > > > > And I see there's a change about Docker file by #10506 (included in
> > > 0.21.0
> > > > > release) which has something to do with the chown command from the
> > > > > discussion.
> > > > >
> > > > > I'm not an expert on Dockfile, hope someone could check this problem.
> > > > >
> > > > >
> > > > > Jihoon Son <jihoon...@apache.org> 于2021年4月27日周二 上午12:56写道：
> > > > >
> > > > > > Frank, thanks for looking into it.
> > > > > >
> > > > > > I'm not sure why it fails in your testing environment. But it seems
> > > > > > like an issue in running the docker image rather than an issue in
> > > > > > the
> > > > > > image itself because it passed tests in other people's environment.
> > > As
> > > > > > a result, it doesn't seem a release blocker to me, so I will go
> > > > > > ahead
> > > > > > and finish this vote. Please file this issue on GitHub so that we
> > > > > > can
> > > > > > track it.
> > > > > >
> > > > > > On Mon, Apr 26, 2021 at 7:16 AM frank chen <frankc...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > I tried to start druid in docker on two MacBookPro(both are
> > > > > > > running
> > > > > > BigSur
> > > > > > > 11.2.3), both failed due to the same reason.
> > > > > > >
> > > > > > > To determine what happened, DRUID_DIRS_TO_CREATE was set in the
> > > > > > environment
> > > > > > > file to skip the directory creation so that druid nodes could
> > > start up,
> > > > > > > and then I checked the directories inside container, found that
> > > owner
> > > > > of
> > > > > > > directory 'var' is root while others are 'druid'.
> > > > > > >
> > > > > > > /opt/apache-druid-0.21.0 # ls -l
> > > > > > >
> > > > > > > total 196
> > > > > > >
> > > > > > > -rw-r--r-- 1 druid druid 70924 Apr 16 02:42 LICENSE
> > > > > > >
> > > > > > > -rw-r--r-- 1 druid druid 71187 Apr 16 02:42 NOTICE
> > > > > > >
> > > > > > > -rw-r--r-- 1 druid druid 8228 Apr 16 02:42 README
> > > > > > >
> > > > > > > drwxr-xr-x 2 druid druid 4096 Apr 16 02:46 *bin*
> > > > > > >
> > > > > > > drwxr-xr-x 5 druid druid 4096 Apr 16 02:46 *conf*
> > > > > > >
> > > > > > > drwxr-xr-x 51 druid druid 4096 Apr 16 02:46
> > > *extensions*
> > > > > > >
> > > > > > > drwxr-xr-x 3 druid druid 4096 Apr 16 02:46
> > > > > > > *hadoop-dependencies*
> > > > > > >
> > > > > > > drwxr-xr-x 2 druid druid 12288 Apr 16 02:46 *lib*
> > > > > > >
> > > > > > > drwxr-xr-x 4 druid druid 4096 Apr 16 02:28
> > > > > > > *licenses*
> > > > > > >
> > > > > > > drwxr-xr-x 4 druid druid 4096 Apr 16 02:46
> > > *quickstart*
> > > > > > >
> > > > > > > drwxr-xr-x 2 root root 4096 Apr 26 13:21 *var*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Since container is running as user 'druid', it has no permission
> > > > > > > to
> > > > > > create
> > > > > > > directories under 'var' directory.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I see the command (see:
> > > > > > >
> > > > > >
> > > > >
> > > https://github.com/apache/druid/blob/0296f205511a2c75f150d978f4cb74757736c54f/distribution/docker/Dockerfile#L51
> > > > > > > )
> > > > > > >
> > > > > > > in Dockerfile has set the owner of all directories to 'druid', but
> > > has
> > > > > no
> > > > > > > idea why owner of 'var' is still 'root'.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I don't know whether this problem happens only on macOS with
> > > > > > > Docker
> > > > > > Desktop
> > > > > > > 3.3.1.
> > > > > > >
> > > > > > > BTW, there's a bug in Docker Desktop 3.3.0 which I sent an email
> > > to the
> > > > > > dev
> > > > > > > last week.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Jihoon Son <jihoon...@apache.org> 于2021年4月24日周六 上午1:27写道：
> > > > > > >
> > > > > > > > Frank, thanks for your testing.
> > > > > > > >
> > > > > > > > Both tests pass on my side.
> > > > > > > > For the dependency check, the NVD database seems back now and
> > > working
> > > > > > > > fine. I sometimes see that the maven dependency check plugin
> > > fails
> > > > > > > > with a false report when you have stale files left over from
> > > previous
> > > > > > > > builds. Can you try again after running 'mvn clean'?
> > > > > > > > For the docker, I'm not sure why those processes could not
> > > > > > > > create
> > > > > > > > directories inside the container. Can you check if there is some
> > > > > > > > permission issue?
> > > > > > > >
> > > > > > > > On Fri, Apr 23, 2021 at 3:43 AM frank chen <frankc...@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Jihoon,
> > > > > > > > >
> > > > > > > > > Here're check results on my environment. And there are 3
> > > problems:
> > > > > > > > > 1) CVE warning
> > > > > > > > > 2) dependency check failure
> > > > > > > > > 3) docker startup failure
> > > > > > > > >
> > > > > > > > > src package:
> > > > > > > > > - verified signature/checksum
> > > > > > > > > - LICENSE/NOTICE present
> > > > > > > > > - CVE check reports vulnerabilities warning as follows
> > > > > > > > > One or more dependencies were identified with known
> > > vulnerabilities
> > > > > > in
> > > > > > > > > druid-core:
> > > > > > > > > commons-io-2.6.jar (pkg:maven/commons-io/commons-io@2.6,
> > > > > > > > > cpe:2.3:a:apache:commons_io:2.6:*:*:*:*:*:*:*) :
> > > > > > > > > CVE-2021-29425
> > > > > > > > > cron-scheduler-0.1.jar
> > > > > (pkg:maven/io.timeandspace/cron-scheduler@0.1
> > > > > > ,
> > > > > > > > > cpe:2.3:a:cron_project:cron:0.1:*:*:*:*:*:*:*) :
> > > > > > > > > CVE-2017-9525,
> > > > > > > > > CVE-2019-9704, CVE-2019-9705
> > > > > > > > > guava-16.0.1.jar (pkg:maven/com.google.guava/guava@16.0.1,
> > > > > > > > > cpe:2.3:a:google:guava:16.0.1:*:*:*:*:*:*:*) : CVE-2018-10237,
> > > > > > > > CVE-2020-8908
> > > > > > > > > hibernate-validator-5.2.5.Final.jar
> > > > > > > > > (pkg:maven/org.hibernate/hibernate-validator@5.2.5.Final,
> > > > > > > > > cpe:2.3:a:hibernate:hibernate-validator:5.2.5:*:*:*:*:*:*:*,
> > > > > > > > > cpe:2.3:a:redhat:hibernate_validator:5.2.5:*:*:*:*:*:*:*) :
> > > > > > > > CVE-2020-10693
> > > > > > > > > log4j-core-2.8.2.jar
> > > > > > (pkg:maven/org.apache.logging.log4j/log4j-core@2.8.2
> > > > > > > > ,
> > > > > > > > > cpe:2.3:a:apache:log4j:2.8.2:*:*:*:*:*:*:*) : CVE-2020-9488
> > > > > > > > > netty-3.10.6.Final.jar (pkg:maven/io.netty/netty@3.10.6.Final,
> > > > > > > > > cpe:2.3:a:netty:netty:3.10.6:*:*:*:*:*:*:*) : CVE-2021-21290,
> > > > > > > > > CVE-2021-21295, CVE-2021-21409
> > > > > > > > > netty-transport-4.1.48.Final.jar
> > > > > > > > > (pkg:maven/io.netty/netty-transport@4.1.48.Final,
> > > > > > > > > cpe:2.3:a:netty:netty:4.1.48:*:*:*:*:*:*:*) : CVE-2021-21290,
> > > > > > > > > CVE-2021-21295, CVE-2021-21409
> > > > > > > > >
> > > > > > > > > - Dependency check failed due to "An error occurred with the
> > > .NET
> > > > > > > > > AssemblyAnalyzer", no more exception message is shown
> > > > > > > > >
> > > > > > > > > binary package:
> > > > > > > > > - verified signature/checksum
> > > > > > > > > - LICENSE, NOTICE and README files are present
> > > > > > > > > - ingested from kafka and ran some queries, and no exception
> > > log
> > > > > > output
> > > > > > > > in
> > > > > > > > > Druid services log files
> > > > > > > > >
> > > > > > > > > docker:
> > > > > > > > > - failed to start cluster with docker-compose.yml in
> > > > > > distribution/docker
> > > > > > > > > directory based on apache/druid:0.21.0-rc1 image, all druid
> > > nodes
> > > > > > > > > unexpected exit with messages like
> > > > > > > > > middlemanager | mkdir: can't create directory 'var/tmp':
> > > > > > Permission
> > > > > > > > > denied
> > > > > > > > > middlemanager | mkdir: can't create directory 'var/druid/':
> > > > > > Permission
> > > > > > > > > denied
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Jihoon Son <jihoon...@apache.org> 于2021年4月17日周六 上午8:59写道：
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I have created a build for Apache Druid 0.21.0, release
> > > > > > > > > > candidate 1.
> > > > > > > > > >
> > > > > > > > > > Thanks for everyone who has helped contribute to the
> > > release! You
> > > > > > can
> > > > > > > > read
> > > > > > > > > > the proposed release notes here:
> > > > > > > > > > https://github.com/apache/druid/issues/10752
> > > > > > > > > >
> > > > > > > > > > The release candidate has been tagged in GitHub as
> > > > > > > > > > druid-0.21.0-rc1 (733697c25ff22045f14016d83b123fa18556dec8),
> > > > > > > > > > available here:
> > > > > > > > > >
> > > https://github.com/apache/druid/releases/tag/druid-0.21.0-rc1
> > > > > > > > > >
> > > > > > > > > > The artifacts to be voted on are located here:
> > > > > > > > > > https://dist.apache.org/repos/dist/dev/druid/0.21.0-rc1/
> > > > > > > > > >
> > > > > > > > > > A staged Maven repository is available for review at:
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > https://repository.apache.org/content/repositories/orgapachedruid-1023/
> > > > > > > > > >
> > > > > > > > > > Staged druid.apache.org website documentation is available
> > > here:
> > > > > > > > > >
> > > https://druid.staged.apache.org/docs/0.21.0/design/index.html
> > > > > > > > > >
> > > > > > > > > > A Docker image containing the binary of the release
> > > candidate can
> > > > > > be
> > > > > > > > > > retrieved via:
> > > > > > > > > > docker pull apache/druid:0.21.0-rc1
> > > > > > > > > >
> > > > > > > > > > artifact checksums
> > > > > > > > > > src:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > 8ff3c5ce96b6eff67a68945284e9d2280ea6fbca4ee4a3a023e74685f05dfbed84d1e9071ed5331cb0b1416cb87895d146ce733ae228070a9437375e1baca022
> > > > > > > > > > bin:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > 4c1b9ff4c8d89e1c78f0bc9e414ea4e855a637925959b5e4e4edd79bdbd0311f0b09cc332c6f48f982f10d9d46d2658cee802bac4e60116598d1aaf3deebf9b1
> > > > > > > > > > docker:
> > > > > > > > 33ff4044017f5974f2e250512a1dd2449078dbf1fa18dd2bd4fa511a4c9f2f78
> > > > > > > > > >
> > > > > > > > > > Release artifacts are signed with the following key:
> > > > > > > > > > https://people.apache.org/keys/committer/jihoonson.asc
> > > > > > > > > >
> > > > > > > > > > This key and the key of other committers can also be found
> > > in the
> > > > > > > > project's
> > > > > > > > > > KEYS file here:
> > > > > > > > > > https://dist.apache.org/repos/dist/release/druid/KEYS
> > > > > > > > > >
> > > > > > > > > > (If you are a committer, please feel free to add your own
> > > key to
> > > > > > that
> > > > > > > > file
> > > > > > > > > > by following the instructions in the file's header.)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Verify checksums:
> > > > > > > > > > diff <(shasum -a512 apache-druid-0.21.0-src.tar.gz | \
> > > > > > > > > > cut -d ' ' -f1) \
> > > > > > > > > > <(cat apache-druid-0.21.0-src.tar.gz.sha512 ; echo)
> > > > > > > > > >
> > > > > > > > > > diff <(shasum -a512 apache-druid-0.21.0-bin.tar.gz | \
> > > > > > > > > > cut -d ' ' -f1) \
> > > > > > > > > > <(cat apache-druid-0.21.0-bin.tar.gz.sha512 ; echo)
> > > > > > > > > >
> > > > > > > > > > Verify signatures:
> > > > > > > > > > gpg --verify apache-druid-0.21.0-src.tar.gz.asc \
> > > > > > > > > > apache-druid-0.21.0-src.tar.gz
> > > > > > > > > >
> > > > > > > > > > gpg --verify apache-druid-0.21.0-bin.tar.gz.asc \
> > > > > > > > > > apache-druid-0.21.0-bin.tar.gz
> > > > > > > > > >
> > > > > > > > > > Please review the proposed artifacts and vote. Note that
> > > Apache
> > > > > has
> > > > > > > > > > specific requirements that must be met before +1 binding
> > > votes
> > > > > can
> > > > > > be
> > > > > > > > cast
> > > > > > > > > > by PMC members. Please refer to the policy at
> > > > > > > > > > http://www.apache.org/legal/release-policy.html#policy for
> > > more
> > > > > > > > details.
> > > > > > > > > >
> > > > > > > > > > As part of the validation process, the release artifacts can
> > > be
> > > > > > > > generated
> > > > > > > > > > from source by running:
> > > > > > > > > > mvn clean install -Papache-release,dist -Dgpg.skip
> > > > > > > > > >
> > > > > > > > > > The RAT license check can be run from source by:
> > > > > > > > > > mvn apache-rat:check -Prat
> > > > > > > > > >
> > > > > > > > > > This vote will be open for at least 72 hours. The vote will
> > > pass
> > > > > > if a
> > > > > > > > > > majority of at least three +1 PMC votes are cast.
> > > > > > > > > >
> > > > > > > > > > [ ] +1 Release this package as Apache Druid 0.21.0
> > > > > > > > > > [ ] 0 I don't feel strongly about it, but I'm okay with the
> > > > > release
> > > > > > > > > > [ ] -1 Do not release this package because...
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > > > > > > > > > For additional commands, e-mail: dev-h...@druid.apache.org
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > > > > > > > For additional commands, e-mail: dev-h...@druid.apache.org
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > > > > > For additional commands, e-mail: dev-h...@druid.apache.org
> > > > > >
> > > > > >
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > > For additional commands, e-mail: dev-h...@druid.apache.org
> > >
> > >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
For additional commands, e-mail: dev-h...@druid.apache.org
