You're right, https://github.com/apache/druid/pull/16386 is merged by me, which was included in 31.0.0. Today I used an earlier release to upgrade from.
* you could get back the earlier behaviour by setting the environment variable `DRUID_SET_HOST_IP=1` - isn't that an option? This environment variable is always set to 0 by the starting script under the K8S environment. How can I reset it? * why is the modification of that `properties` out-of-scope - can't you remove druid.host=localhost from it ; and where its coming from? It's from the common.runtime.properties file, how can I remove it? According to current script behaviour, I can set the druid_host to override it, but I think this is not the correct way, why should we introduce such a change to make the upgrade so difficult? * is this deployment follows some playbook which should be supported out-of-the-box? if yes; could you point me to it? We don't use druid operator, but use the helm-chart ( https://github.com/asdf2014/druid-helm) I assume that you're using this one: https://github.com/druid-io/druid-operator/blob/master/examples/tiny-cluster.yaml As you can see from the druid-operator, it mounts a customized common.runtime.properties file(which does not include the druid.host property) to override the default one(which has the druid.host property) which is provided by the image. This is why the problem is not reproduced in your environment. On Thu, Apr 17, 2025 at 9:49 PM Zoltan Haindrich <k...@rxd.hu> wrote: > Hey Frank, > > I've deployed the apache/druid:32.0.0-rc2 image on minikube with > druid-operator and it has fully qualified hostnames in zk [1] > So it seems to me that it works correctly - even in case there is another > cluster in another namespace using the same ZK I guess that should work too. > > While I was digginig into history: > there was > * Use canonical hostname instead of ip by default > https://github.com/apache/druid/pull/16386 > * which was merged by you - and it essentially made the same changes as > you see right now on k8s > * reverted by https://github.com/apache/druid/pull/17347 > * reapplied with restriction to only affect k8s in > https://github.com/apache/druid/pull/17697 > > > To be sure to get on the same page; could you answer these questions: > * you could get back the earlier behaviour by setting the environment > variable `DRUID_SET_HOST_IP=1` - isn't that an option? > * why is the modification of that `properties` out-of-scope - can't you > remove druid.host=localhost from it ; and where its coming from? > * is this deployment follows some playbook which should be supported > out-of-the-box? if yes; could you point me to it? > > > cheers, > Zoltan > > > [1] > $ echo 'ls -R /' | kubectl exec -it pod/tiny-cluster-zk-0 zkCli.sh > [...] > > /druid/internal-discovery/BROKER/druid-tiny-cluster-brokers-0.druid-tiny-cluster-brokers.druid.svc.cluster.local:8088 > > /druid/internal-discovery/COORDINATOR/druid-tiny-cluster-coordinators-0.druid-tiny-cluster-coordinators.druid.svc.cluster.local:8088 > > /druid/internal-discovery/HISTORICAL/druid-tiny-cluster-historicals-0.druid-tiny-cluster-historicals.druid.svc.cluster.local:8088 > > /druid/internal-discovery/OVERLORD/druid-tiny-cluster-coordinators-0.druid-tiny-cluster-coordinators.druid.svc.cluster.local:8088 > > /druid/internal-discovery/ROUTER/druid-tiny-cluster-routers-0.druid-tiny-cluster-routers.druid.svc.cluster.local:8088 > [...] > > > > On 4/17/25 12:52, Frank Chen wrote: > > In the previous release, druid.host is SET by the starting script to ip > > address, > > now, since KUBERNETES_SERVICE_HOST environment variable is detected in > the > > K8S env, the druid.host will NOT be set, and the localhost defined in the > > property file will be used. > > > > I think druid_host should be allowed to be customized instead of fully > > controlled by our script. > > One case is that if a druid cluster is deployed in two K8S clusters, > > services in two K8S clusters can't communicate with each other by the > > default host name. we need to add some suffix to the host name to use > > CoreDNS to resolve the druid_host. > > > > > > > > On Thu, Apr 17, 2025 at 4:46 PM Zoltan Haindrich <k...@rxd.hu> wrote: > > > >> Hey Frank, > >> > >> Thank you for checking it out - I was digging into it a bit ; but I'm > not > >> sure if I understand all aspects of your setup/expectations > >> If you were you following a guide for k8s - could you point to it; it > >> might help a lot to get on the same page! > >> > >> Let me try to condense how I think this works right now: > >> > >> * druid.host might be set in a properties file > >> * depending on the values of `KUBERNETES_SERVICE_HOST` and > >> `DRUID_SET_HOST_IP` it might be set in a script > >> * default behaviour for k8s seems to be to not set `druid.host` to > >> anything > >> * if it remains unset it will be null and populated in DruidNode [1] to > >> `InetAddress.getLocalHost().getCanonicalHostName()` > >> > >> I believe for a k8s deployment you should not set druid.host to anything > >> and let `DruidNode` set its hostname. > >> > >> I suspect earlier it worked because it was set to the ip address on k8s > as > >> well - regardless what the config contained; I believe hostnames are > much > >> friendlier to use - so > >> it would be great to use them. > >> > >> Regarding the alternatives: > >> > 1. if the druid_host environment is not provided, use the hostname > >> command > >> > to set the druid.host property if it's under the K8S environment to > >> > override the default one in the property file > >> > >> I see that there is a `druid_host` env [2] ; but instead of adding more > >> custom scripts I would rather leave that to > >> `InetAddress.getLocalHost().getCanonicalHostName()` if > >> possible. > >> > >> > 2. or remove the druid.host default property from the file and > always > >> set > >> > it via starting scripts > >> > >> I think you mean to remove it from > >> conf/druid/cluster/_common/common.runtime.properties [3] ? > >> But I feel like that might be done automatically people following: > >> https://druid.apache.org/docs/latest/tutorials/cluster/ > >> opened PR to unset it: https://github.com/apache/druid/pull/17930 > >> > >> > BTW, I think we should add this change(use the hostname under the > K8S > >> > environment) to the release note too. > >> added a suggestion to the release notes PR relating to this. > >> > >> > >> cheers, > >> Zoltan > >> > >> [1] > >> > https://github.com/apache/druid/blob/3766e3a14fb49ca98ab3b02495bf1336c7eee75c/server/src/main/java/org/apache/druid/server/DruidNode.java#L168 > >> [2] > >> > >> > https://github.com/apache/druid/blob/3766e3a14fb49ca98ab3b02495bf1336c7eee75c/extensions-core/kubernetes-overlord-extensions/src/main/java/org/apache/druid/k8s/overlord/common/DruidK8sConstants.java#L43 > >> [3] > >> > https://github.com/apache/druid/blob/3766e3a14fb49ca98ab3b02495bf1336c7eee75c/examples/conf/druid/cluster/_common/common.runtime.properties#L37 > >> > >> > >> On 4/17/25 05:37, Frank Chen wrote: > >>> -1(binding) because https://github.com/apache/druid/pull/17697 > prevents > >> us > >>> from upgrading our old clusters to this release. > >>> > >>> Our clusters are running in K8S, above PR skips the setting of > druid.host > >>> properties, however, this property is defined as localhost by default > in > >>> the common.properties, > >>> so after upgrading, all druid service names registered in the zookeeper > >> are > >>> localhost, which can't be accessed by other services. > >>> > >>> Possible solutions might be: > >>> 1. if the druid_host environment is not provided, use the hostname > >> command > >>> to set the druid.host property if it's under the K8S environment to > >>> override the default one in the property file > >>> 2. or remove the druid.host default property from the file and always > set > >>> it via starting scripts > >>> > >>> BTW, I think we should add this change(use the hostname under the K8S > >>> environment) to the release note too. > >>> > >>> > >>> > >>> > >>> On Thu, Apr 17, 2025 at 8:25 AM Clint Wylie <cwy...@apache.org> wrote: > >>> > >>>> +1 (binding) > >>>> > >>>> src package: > >>>> * verified checksum and signature > >>>> * checked for LICENSE and NOTICE > >>>> * rat check successful > >>>> * built binary distribution, ran MSQ ingestion, kafka ingestion, > >>>> compaction, then some queries > >>>> > >>>> binary package: > >>>> * verified checksum and signature > >>>> * checked for LICENSE and NOTICE > >>>> * ran json batch ingestion and some queries > >>>> > >>>> docker: > >>>> * verified checksum > >>>> * started cluster with docker-compose, ran native json ingestion, some > >>>> queries > >>>> > >>>> I did see an out of memory failure on coordinator-overlord when using > >>>> the old 'micro quickstart' configuration when I was testing kafka > >>>> ingestion... I would guess it is related to the sampler? However, I > >>>> did not look much further into it since I think the recommended way to > >>>> run tutorials these days is with the auto sizing script, so not sure > >>>> if it is a big deal or not. Did not see the problem when using that > >>>> script instead. > >>>> > >>>> On Wed, Apr 16, 2025 at 12:39 AM Zoltan Haindrich <k...@rxd.hu> > wrote: > >>>>> > >>>>> Hi all, > >>>>> > >>>>> I have created a build for Apache Druid 33.0.0, release candidate 2. > >>>> (RC1 was botched). > >>>>> > >>>>> Thanks for everyone who has helped contribute to the release! You can > >>>> read > >>>>> the proposed release notes here: > >>>>> https://github.com/apache/druid/pull/17872 > >>>>> > >>>>> The release candidate has been tagged in GitHub as druid-33.0.0-rc2 > >>>> (63477ea1b7866dc2f61c9d1d893dea150735cbff), > >>>>> available here: > https://github.com/apache/druid/tree/druid-33.0.0-rc2 > >>>>> > >>>>> The artifacts to be voted on are located here: > >>>>> https://dist.apache.org/repos/dist/dev/druid/33.0.0-rc2/ > >>>>> > >>>>> A staged Maven repository is available for review at: > >>>>> > >> https://repository.apache.org/content/repositories/orgapachedruid-1086/ > >>>>> > >>>>> Staged druid.apache.org website documentation is available here: > >>>>> https://druid.staged.apache.org/docs/33.0.0/ > >>>>> > >>>>> A Docker image containing the binary of the release candidate can be > >>>>> retrieved via: > >>>>> docker pull apache/druid:33.0.0-rc2 > >>>>> > >>>>> artifact checksums: > >>>>> 33.0.0-rc2/apache-druid-33.0.0-bin.tar.gz > >>>>> > >>>> > >> > cd6adc2840c87521312534cd6a786b0d92559a652f275a6d773e717cef1c5bd0f6aacac67626e2a26fdad96f2353c5314e53b8a2533b37cbdc953d96025a1e04 > >>>>> > >>>>> apache-druid-33.0.0-src.tar.gz > >>>>> > >>>> > >> > d4f78b627e0c46db0d328ce53460ae3f5639831f20efc50ed6c25c5999aa310dd089d4d7813c0994f7c6714a2df24594ab7b14ab087e330bb2fbc5992e6e7293 > >>>>> > >>>>> docker: > >>>> > sha256:528a78a68b1dde47a5984e60975203d8cdf0bd4f477f8b97b84a8b2b44e3c683 > >>>>> > >>>>> > >>>>> Release artifacts are signed with the following key: > >>>>> https://people.apache.org/keys/committer/kgyrtkirk.asc > >>>>> > >>>>> This key and the key of other committers can also be found in the > >>>> project's > >>>>> KEYS file here: > >>>>> https://dist.apache.org/repos/dist/release/druid/KEYS > >>>>> > >>>>> (If you are a committer, please feel free to add your own key to that > >>>> file > >>>>> by following the instructions in the file's header.) > >>>>> > >>>>> Verify checksums: > >>>>> diff <(shasum -a512 apache-druid-33.0.0-src.tar.gz | cut -d ' ' -f1) > >>>> <(cat apache-druid-33.0.0-src.tar.gz.sha512 ; echo) > >>>>> diff <(shasum -a512 apache-druid-33.0.0-bin.tar.gz | cut -d ' ' -f1) > >>>> <(cat apache-druid-33.0.0-bin.tar.gz.sha512 ; echo) > >>>>> > >>>>> Verify signatures: > >>>>> gpg --verify apache-druid-33.0.0-src.tar.gz.asc > >>>> apache-druid-33.0.0-src.tar.gz > >>>>> gpg --verify apache-druid-33.0.0-bin.tar.gz.asc > >>>> apache-druid-33.0.0-bin.tar.gz > >>>>> > >>>>> Please review the proposed artifacts and vote. Note that Apache has > >>>>> specific requirements that must be met before +1 binding votes can be > >>>> cast > >>>>> by PMC members. Please refer to the policy at > >>>>> http://www.apache.org/legal/release-policy.html#policy for more > >> details. > >>>>> > >>>>> As part of the validation process, the release artifacts can be > >> generated > >>>>> from source by running: > >>>>> mvn clean install -Papache-release,dist -Dgpg.skip > >>>>> > >>>>> The RAT license check can be run from source by: > >>>>> mvn apache-rat:check -Prat > >>>>> > >>>>> This vote will be open for at least 72 hours. The vote will pass if a > >>>>> majority of at least three +1 PMC votes are cast. > >>>>> > >>>>> [ ] +1 Release this package as Apache Druid 0.17.0 > >>>>> [ ] 0 I don't feel strongly about it, but I'm okay with the release > >>>>> [ ] -1 Do not release this package because... > >>>>> > >>>>> cheers, > >>>>> Zoltan > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > >>>> For additional commands, e-mail: dev-h...@druid.apache.org > >>>> > >>>> > >>> > >> > >> > > > >
