I just tried the job and it worked as expected. Thanks! If I understand correctly, the job retains builds for 180 days. Does it mean if there were no commits to a branch for more than 180 days, we will lose the build artifacts eventually?
On Thu, May 25, 2023 at 1:50 AM Zoltan Haindrich <k...@rxd.hu> wrote: > Hey Vihang, > > I've added you as an admin; and I've copied the job as > http://ci.hive.apache.org/job/hive-nightly-branch-3/ > other option could be to trigger the original job or use > parameterized-scheduler but that would configure a real unconditional > nightly build - which will just build the > same version over-and-over again if there are no changes... > ...the current nighly is SCM triggered ; but only once-a-day it makes a > check which creates the desired results. > > the least painfull was to copy the job; I guess no-one touched the > pipeline script ever since it was introduced :D > > cheers, > Zoltan > > On 5/25/23 01:26, vihang karajgaonkar wrote: > > I created https://issues.apache.org/jira/browse/HIVE-27371 to have > nightly > > builds for branch-3. Once that is merged, I think we can have scheduled > > builds for branch-3 as well. Although, I don't have permissions to > create a > > new job for branch-3. Does anyone know how to do it? > > > > Thanks, > > Vihang > > > > On Wed, May 24, 2023 at 10:07 AM vihang karajgaonkar < > vihan...@apache.org> > > wrote: > > > >> The nightly job http://ci.hive.apache.org/job/hive-nightly/ is great. > Can > >> we have this for branch-3 as well since we have been backporting a lot > of > >> PRs to branch-3 lately. > >> > >> Thanks, > >> Vihang > >> > >> > >> > >> > >> > >> On Wed, May 24, 2023 at 6:56 AM Zoltan Haindrich <k...@rxd.hu> wrote: > >> > >>> Hey, > >>> > >>> > We already have nightly builds for Hive [1]. > >>> > [1] http://ci.hive.apache.org/job/hive-nightly/ > >>> > >>> ...and hive-dev-box can launch such archives; either by using it like > >>> this: > >>> https://www.mail-archive.com/dev@hive.apache.org/msg142420.html > >>> > >>> or with a somewhat longer command you could launch hdb in bazaar mode; > >>> and have an HS2 running with a nightly version: > >>> > >>> docker run --rm -d -p 10000:10000 -v hive-dev-box_work:/work -e > >>> HIVE_VERSION= > >>> > http://ci.hive.apache.org/job/hive-nightly/lastSuccessfulBuild/artifact/archive/apache-hive-4.0.0-nightly-b0b3fde70c-20230524_014711-bin.tar.gz > >>> --name hive > >>> kgyrtkirk/hive-dev-box:bazaar > >>> > >>> cheers, > >>> Zoltan > >>> > >>> On 5/24/23 09:15, Stamatis Zampetakis wrote: > >>>> Hey all, > >>>> > >>>> We already have nightly builds for Hive [1]. > >>>> > >>>> Do we need something more than that? > >>>> > >>>> Best, > >>>> Stamatis > >>>> > >>>> [1] http://ci.hive.apache.org/job/hive-nightly/ > >>>> > >>>> > >>>> On Tue, May 23, 2023 at 9:03 AM vihang karajgaonkar < > >>> vihan...@apache.org> wrote: > >>>>> > >>>>> I think there are many benefits like others in this thread suggested > >>> which > >>>>> can be built on top of nightly builds. Having docker images is great > >>> but > >>>>> for now I think we can start simple and publish the jars. Many users > >>> still > >>>>> just deploy using jars and it would be useful to them. Once we have a > >>>>> docker environment we can add a docker image too to the nightly > builds > >>> so > >>>>> that users can choose their preferred way. > >>>>> > >>>>> On Mon, May 22, 2023 at 11:07 PM Sungwoo Park <glap...@gmail.com> > >>> wrote: > >>>>> > >>>>>> I think such nightly builds will be useful for testing and debugging > >>> in the > >>>>>> future. > >>>>>> > >>>>>> I also wonder if we can somehow create builds even from previous > >>> commits > >>>>>> (e.g., for the past few years). Such builds from previous commits > >>> don't > >>>>>> have to be daily builds, and I think weekly builds (or even monthly > >>> builds) > >>>>>> would also be very useful. > >>>>>> > >>>>>> The reason I wish such builds were available is to facilitate > >>> debugging and > >>>>>> testing. When tested against the TPC-DS benchmark, the current > master > >>>>>> branch has several correctness problems that were introduced after > the > >>>>>> release of Hive 3.1.2. We have reported all problems known to us in > >>> [1] and > >>>>>> also submitted several patches. If such nightly builds had been > >>> available, > >>>>>> we would have saved quite a bit of time for implementing the patches > >>> by > >>>>>> quickly finding offending commits that introduced new correctness > >>> bugs. > >>>>>> > >>>>>> In addition, you can find quite a few commits in the master branch > >>> that > >>>>>> report bugs which are not reproduced in Hive 3.1.2. Examples: > >>> HIVE-19990, > >>>>>> HIVE-14557, HIVE-21132, HIVE-21188, HIVE-21544, HIVE-22114, > >>>>>> HIVE-22227, HIVE-22236, HIVE-23911, HIVE-24198, HIVE-22777, > >>>>>> HIVE-25170, HIVE-25864, HIVE-26671. > >>>>>> (There may be some errors in this list because we compared against > >>> Hive > >>>>>> 3.1.2 with many patches backported.) Such nightly builds can be > >>> useful for > >>>>>> finding root causes of such bugs. > >>>>>> > >>>>>> Ideally I wish there was an automated procedure to create nightly > >>> builds, > >>>>>> run TPC-DS benchmark, and report correctness/performance results, > >>> although > >>>>>> this would be quite hard to implement. (I remember Spark implemented > >>> this > >>>>>> procedure in the era of Spark 2, but my memory could be wrong.) > >>>>>> > >>>>>> [1] https://issues.apache.org/jira/browse/HIVE-26654 > >>>>>> > >>>>>> > >>>>>> On Tue, May 23, 2023 at 10:44 AM Ayush Saxena <ayush...@gmail.com> > >>> wrote: > >>>>>> > >>>>>>> Hi Vihang, > >>>>>>> +1, We were even exploring publishing the docker images of the > >>> snapshot > >>>>>>> version as well per commit or maybe weekly, so just shoot 2 docker > >>>>>> commands > >>>>>>> and you get a Hive cluster running with master code. > >>>>>>> > >>>>>>> Sai, I think to spin up an env via Docker with all these things > >>> should be > >>>>>>> doable for sure, but would require someone with real good expertise > >>> with > >>>>>>> docker as well as setting up these services with Hive. Obviously, I > >>> am > >>>>>> not > >>>>>>> that guy :-) > >>>>>>> > >>>>>>> @Simhadri has a PR which publishes docker images once a release tag > >>> is > >>>>>>> pushed, you can explore to have similar stuff for the Snapshot > >>> version, > >>>>>>> maybe if that sounds cool > >>>>>>> > >>>>>>> -Ayush > >>>>>>> > >>>>>>> On Tue, 23 May 2023 at 04:26, Sai Hemanth Gantasala > >>>>>>> <saihema...@cloudera.com.invalid> wrote: > >>>>>>> > >>>>>>>> Hi Vihang, > >>>>>>>> > >>>>>>>> +1 on the idea. > >>>>>>>> > >>>>>>>> This is a great idea to quickly test if a certain feature is > >>> working as > >>>>>>>> expected on a certain branch. > >>>>>>>> This way we test data loss, correctness, or any other unexpected > >>>>>>> scenarios > >>>>>>>> that are Hive specific only. However, I'm wondering if it is > >>> possible > >>>>>> to > >>>>>>>> deploy/test in a kerberized environment or issues involving > >>>>>> authorization > >>>>>>>> services like sentry/ranger. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Sai. > >>>>>>>> > >>>>>>>> On Mon, May 22, 2023 at 11:15 AM vihang karajgaonkar < > >>>>>>> vihan...@apache.org> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hello Team, > >>>>>>>>> > >>>>>>>>> I have observed that it is a common use-case where users would > like > >>>>>> to > >>>>>>>> test > >>>>>>>>> out unreleased features/bug fixes either to unblock them or test > >>> out > >>>>>> if > >>>>>>>> the > >>>>>>>>> bug fixes really work as intended in their environments. Today in > >>> the > >>>>>>>> case > >>>>>>>>> of Apache Hive, this is not very user friendly because it > requires > >>>>>> the > >>>>>>>> end > >>>>>>>>> user to build the binaries directly from the hive source code. > >>>>>>>>> > >>>>>>>>> I found that Apache Spark has a very useful infrastructure [1] > >>> which > >>>>>>>>> deploys nightly snapshots [2] [3] from the branch using github > >>>>>> actions. > >>>>>>>>> This is super useful for any user who wants to try out the latest > >>> and > >>>>>>>>> greatest using the nightly builds. > >>>>>>>>> > >>>>>>>>> I was wondering if we should also adopt this. We can use github > >>>>>> actions > >>>>>>>> to > >>>>>>>>> upload the snapshot jars to the public repository (e.g github > >>>>>> packages) > >>>>>>>> and > >>>>>>>>> schedule it as a nightly job. > >>>>>>>>> > >>>>>>>>> [1] https://issues.apache.org/jira/browse/INFRA-21167 > >>>>>>>>> [2] > >>>>>>> > https://github.com/apache/spark/pkgs/container/apache-spark-ci-image > >>>>>>>>> [3] https://github.com/apache/spark/pull/30623 > >>>>>>>>> > >>>>>>>>> I can take a stab at this if the community thinks that this is a > >>> nice > >>>>>>>> thing > >>>>>>>>> to have. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Vihang > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>> > >> > > >
