Hi Team,
Could we create tickets for the issues?
I think it would be good to collect the issues/potential blockers in the
jira instead of having a complicated mail thread.
If we set the target version to 4.0.0-alpha-1, then we can easily use the
following filter to see the status of the tasks:
https://issues.apache.org/jira/issues/?jql=project%3D%22HIVE%22%20AND%20%22Target%20Version%2Fs%22%3D%224.0.0-alpha-1%22
<
https://issues.apache.org/jira/issues/?jql=project=%22HIVE%22%20AND%20%22Target%20Version/s%22=%224.0.0-alpha-1%22
@Stamatis: Sadly I have missed your letter/jira and created my own with
the fix for building from the src package:
https://issues.apache.org/jira/browse/HIVE-25997 <
https://issues.apache.org/jira/browse/HIVE-25997>
If you have time, I would like to ask you to review.
If anyone knows of any blocker I would like to ask them to create a jira
for that too.
Thanks,
Peter
On 2022. Mar 2., at 7:04, Sungwoo Park <c...@pl.postech.ac.kr> wrote:
Hello Alessandro,
For the latest commit, loading ORC tables fails (with the log message
shown below). Let me try to find a commit that introduces this bug and
create a JIRA ticket.
--- Sungwoo
2022-03-02 05:41:56,578 ERROR [Thread-73] exec.StatsTask: Failed to run
stats task
java.io.IOException: org.apache.hadoop.mapred.InvalidInputException:
Input path does not exist:
hdfs://blue0:8020/tmp/hive/gitlab-runner/a236e1b4-b354-4343-b900-3d71b1bc7504/hive_2022-03-02_05-40-50_966_446574755576325031-1/-mr-10000/.hive-staging_hive_2022-03-02_05-40-50_966_446574755576325031-1/-ext-10001
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:622)
at
org.apache.hadoop.hive.ql.stats.ColStatsProcessor.constructColumnStatsFromPackedRows(ColStatsProcessor.java:105)
at
org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:200)
at
org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:93)
at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:107)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:83)
Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path
does not exist:
hdfs://blue0:8020/tmp/hive/gitlab-runner/a236e1b4-b354-4343-b900-3d71b1bc7504/hive_2022-03-02_05-40-50_966_446574755576325031-1/-mr-10000/.hive-staging_hive_2022-03-02_05-40-50_966_446574755576325031-1/-ext-10001
at
org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:294)
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236)
at
org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:435)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:402)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:306)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:560)
... 7 more
On Tue, 1 Mar 2022, Alessandro Solimando wrote:
Hi Sungwoo,
last time I tried to run TPCDS-based benchmark I stumbled upon a similar
situation, finally I found that statistics were not computed, so CBO was
not kicking in, and the automatic retry goes with CBO off which was
failing
for something like 10 queries (subqueries cannot be decorrelated, but
also
some runtime errors).
Making sure that (column) statistics were correctly computed fixed the
problem.
Can you check if this is the case for you?
HTH,
Alessandro
On Tue, 1 Mar 2022 at 15:28, POSTECH CT <c...@pl.postech.ac.kr> wrote:
Hello Hive team,
I wonder if anyone in the Hive team has tried the TPC-DS benchmark on
the master branch recently. We occasionally run TPC-DS system tests
using the master branch, and the tests don't succeed completely. Here
is how our TPC-DS tests proceed.
1. Compile and run Hive on Tez (not Hive-LLAP)
2. Load ORC tables from 1TB TPC-DS raw text data, and compute
statistics
3. Run 99 TPC-DS queries which were slightly modified to return
varying number of rows (rather than 100 rows)
4. Compare the results against the previous results
The previous results were obtained and cross-checked by running Hive
3.1.2 and SparkSQL 2.3/3.2, so we are faily confident about their
correctness.
For the latest commit in the master branch, step 2 fails. For earlier
commits (for example, commits in February 2021), step 3 fails where
several queries either fail or return wrong results.
We can compile and report the test results in this mailing list, but
would like to know if similar results have been reproduced by the Hive
team, in order to make sure that we did not make errors in our tests.
If it is okay to open a JIRA ticket that only reports failures in the
TPC-DS test, we could also perform git bi-sect to locate the commit
that begin to generate wrong results.
--- Sungwoo Park
On Tue, 1 Mar 2022, Zoltan Haindrich wrote:
Hey,
Great to hear that we are on the same side regarding these things :)
For around a week now - we have nightly builds for the master branch:
http://ci.hive.apache.org/job/hive-nightly/12/
I think we have 1 blocker issue:
https://issues.apache.org/jira/browse/HIVE-25665
I know about one more thing I would rather get fixed before we release
it:
https://issues.apache.org/jira/browse/HIVE-25994
The best would be to introduce smoke tests (HIVE-22302) to ensure that
something like this will not happen in the future - but we should
probably
start moving forward.
I think we could call the first iteration of this as "4.0.0-alpha-1"
:)
I've added 4.0.0-alpha-1 as a version - and added the above two ticket
to it.
https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20fixVersion%20%3D%204.0.0-alpha-1
Are there any more things you guys know which would be needed?
cheers,
Zoltan
On 2/22/22 12:18 PM, Peter Vary wrote:
I would vote for 4.0.0-alpha-1 or similar for all of the components.
When we have more stable releases I would keep the 4.x.x schema,
since
everyone is familiar with it, and I do not see a really good reason
to
change it.
Thanks,
Peter
On 2022. Feb 10., at 3:34, Szehon Ho <szehon.apa...@gmail.com>
wrote:
+1 that would be awesome to see Hive master released after so long.
Either 4.0 or 4.0.0-alpha-1 makes sense to me, not sure how we would
pick
any 3.x or calendar date (which could tend to slip and be more
confusing?).
Thanks in any case to get the ball rolling.
Szehon
On Wed, Feb 9, 2022 at 4:55 AM Zoltan Haindrich <k...@rxd.hu>
wrote:
Hey,
Thank you guys for chiming in; versioning is for sure something we
should
get to some common ground.
Its a triple problem right now; I think we have the following
things:
* storage-api
** we have "2.7.3-SNAPSHOT" in the repo
***
https://github.com/apache/hive/blob/0d1cffffc7c5005fe47759298fb35a1c67edc93f/storage-api/pom.xml#L27
** meanwhile we already have 2.8.1 released to maven central
***
https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api
* standalone-metastore
** 4.0.0-SNAPSHOT in the repo
** last release is 3.1.2
* hive
** 4.0.0-SNAPSHOT in the repo
** last release is 3.1.2
Regarding the actual version number I'm not entirely sure where we
should
start the numbering - that's why I was referring to it as Hive-X
in my
first letter.
I think the key point here would be to start shipping releases
regularily
and not the actual version number we will use - I'll kinda open to
any
versioning scheme which
reflects that this is a newer release than 3.1.2.
I could imagine the following ones:
(A) start with something less expected; but keep 3 in the prefix to
reflect that this is not yet 4.0
I can imagine the following numbers:
3.900.0, 3.901.0, ...
3.9.0, 3.9.1, ...
(B) start 4.0.0
4.0.0, 4.1.0, ...
(C) jump to some calendar based version number like 2022.2.9
trunk based development has pros and cons...making a move like
this
irreversibly pledges trunk based development; and makes release
branches
hard to introduce
(X) somewhat orthogonal is to (also) use some suffixes
4.0.0-alpha1, 4.0.0-alpha2, 4.0.0-beta1
this is probably the most tempting to use - but this versioning
schema with a non-changing MINOR and PATCH number will
also suggest that the actual software is fully compatible - and
only
bugs are being fixed - which will not be true...
I really like the idea to suffix these releases with alpha or beta
-
which
will communicate our level commitment that these are not 100%
production
ready artifacts.
I think we could fix HIVE-25665; and probably experiment with
4.0.0-alpha1
for start...
This also means there should *not* be a branch-4 after releasing
Hive
4.0
and let that diverge (and becomes the next, super-ignored
branch-3),
correct; no need to keep a branch we don't maintain...but in any
case
I
think we can postpone this decision until there will be something
to
release... :)
cheers,
Zoltan
On 2/9/22 10:23 AM, L?szl? Bodor wrote:
Hi All!
A purely technical question: what will the SNAPSHOT version become
after
releasing Hive 4.0.0? I think this is important, as it defines and
reflects
the future release plans.
Currently, it's 4.0.0-SNAPSHOT, I guess it's since Hive 3.0 +
branch-3.
Hive is an evolving and super-active project: if we want to make
regular
releases, we should simply release Hive 4.0 and bump pom to
4.1.0-SNAPSHOT,
which clearly says that we can release Hive 4.1 anytime we want,
without
being frustrated about "whether we included enough cool stuff to
release
5.0".
This also means there should *not* be a branch-4 after releasing
Hive
4.0
and let that diverge (and becomes the next, super-ignored
branch-3),
only
when we end up bringing a minor backward-incompatible thing that
needs a
4.0.x, and when it happens, we'll create *branch-4.0 *on demand.
For
me,
a
branch called *branch-4.0* doesn't imply either I can expect cool
releases
in the future from there or the branch is maintained and tries to
be
in
sync with the *master*.
Regards,
Laszlo Bodor
Alessandro Solimando <alessandro.solima...@gmail.com> ezt ?rta
(id?pont:
2022. febr. 8., K, 16:42):
Hello everyone,
thank you for starting this discussion.
I agree that releasing the master branch regularly and
sufficiently
often
is welcome and vital for the health of the community.
It would be great to hear from others too, especially PMC members
and
committers, but even simple contributors/followers as myself.
Best regards,
Alessandro
On Wed, 2 Feb 2022 at 12:22, Stamatis Zampetakis <
zabe...@gmail.com
wrote:
Hello,
Thanks for starting the discussion Zoltan.
I strongly believe that it is important to have regular and
often
releases
otherwise people will create and maintain separate Hive forks.
The latter is not good for the project and the community may
lose
valuable
members because of it.
Going forward I fully agree that there is no point bringing up
strong
blockers for the next release. For sure there are many backward
incompatible changes and possibly unstable features but unless
we
get
a
release out it will be difficult to determine what is broken and
what
needs
to be fixed.
Due to the big number of changes that are going to appear in the
next
version I would suggest using the terms Hive X-alpha, Hive
X-beta
for
the
first few releases. This will make it clear to the end users
that
they
need
to be careful when upgrading from an older version and it will
give us
a
bit more time and freedom to treat issues that the users will
likely
discover.
The only real blocker that we may want to treat is HIVE-25665
[1]
but
we
can continue the discussion under that ticket and re-evaluate if
necessary,
Best,
Stamatis
[1] https://issues.apache.org/jira/browse/HIVE-25665
On Tue, Feb 1, 2022 at 5:03 PM Zoltan Haindrich <k...@rxd.hu>
wrote:
Hey All,
We didn't made a release for a long time now; (3.1.2 was
released
on
26
August 2019) - and I think because we didn't made that many
branch-3
releases; not too many fixes
were ported there - which made that release branch kinda erode
away.
We have a lot of new features/changes in the current master.
I think instead of aiming for big feature-packed releases we
should
aim
for making a regular release every few months - we should make
regular
releases which people could
install and use.
After all releasing Hive after more than 2 years would be big
step
forward
in itself alone - we have so many improvements that I can't
even
count...
But I may know not every aspects of the project / states of
some
internal
features - so I would like to ask you:
What would be the bare minimum requirements before we could
release
the
current master as Hive X?
There are many nice-to-have-s like:
* hadoop upgrade
* jdk11
* remove HoS or MR
* ?
but I don't think these are blockers...we can make any of these
in
the
next release if we start making them...
cheers,
Zoltan
