[GitHub] hive pull request #350: HIVE-19485 : dump directory for non native tables sh...

[anishek]

GitHub user anishek opened a pull request:

https://github.com/apache/hive/pull/350

HIVE-19485 : dump directory for non native tables should not be created



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anishek/hive HIVE-19485

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/350.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #350


commit 83d9e701bb317c989372d2c523912cc0cc1e9eae
Author: Anishek Agarwal 
Date:   2018-05-10T10:37:26Z

HIVE-19485 : dump directory for non native tables should not be created




---

[jira] [Created] (HIVE-19569) alter table db1.t1 rename db2.t2 generates MetaStoreEventListener.onDropTable()

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-19569:
-

 Summary: alter table db1.t1 rename db2.t2 generates 
MetaStoreEventListener.onDropTable()
 Key: HIVE-19569
 URL: https://issues.apache.org/jira/browse/HIVE-19569
 Project: Hive
  Issue Type: Bug
  Components: Metastore, Standalone Metastore, Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman


When renaming a table within the same DB, this operation causes 
{{MetaStoreEventListener.onAlterTable()}} to fire but when changing DB name for 
a table it causes {{MetaStoreEventListener.onDropTable()}} + 
{{MetaStoreEventListener.onCreateTable()}}.
The files from original table are moved to new table location.  
This creates confusing semantics since any logic in {{onDropTable()}} doesn't 
know about the larger context, i.e. that there will be a matching 
{{onCreateTable()}}.

In particular, this causes a problem for Acid tables since files moved from old 
table use WriteIDs that are not meaningful with the context of new table.

Current implementation is due to replication.  This should ideally be changed 
to raise a "not supported" error for tables that are marked for replication.

cc [~sankarh]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Unsustainable situation with ptests

[Siddharth Seth]

Very nice. There was an effort to get fast and green builds back in 2016.
There wasn't any strict "must be a green build" before commit at the time
though. Instead jiras were filed and the expectation was that they'd be
cited / new ones created pre commit(looking at the jiras now - this was
likely followed for a while, many fixes, and eventually got annoying?).
Think the enforcement step is absolutely required to get to, and maintain a
green build. May want to consider performance characteristics of tests as
well - must complete with X seconds.
Jiras for reference (including test infra improvements which were not done
at the time): HIVE-13503, HIVE-15058, HIVE-14547

This will be painful initially, but eventually it'll be great to be able to
commit without having to scan through a bunch of 'known failures', analyze,
document etc.

On Tue, May 15, 2018 at 5:30 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> Wow! Awesome. This is the 3rd time I remember seeing green run in >4yrs. :)
>
> Thanks
> Prasanth
>
> > On May 15, 2018, at 5:28 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
> >
> > We have just had the first clean run in a while:
> > https://builds.apache.org/job/PreCommit-HIVE-Build/10971/testReport/
> >
> > I will continue monitoring follow-up runs.
> >
> > Thanks,
> > -Jesús
> >
> >
> > On 5/14/18, 11:28 PM, "Prasanth Jayachandran" <
> pjayachand...@hortonworks.com> wrote:
> >
> >Wondering if we can add a state transition from “Patch Available” to
> “Ready To Commit” which can only be triggered by ptest bot on green test
> run.
> >
> >Thanks
> >Prasanth
> >
> >
> >
> >On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
> jcama...@apache.org> wrote:
> >
> >
> >I have been working on fixing this situation while commits were still
> coming in.
> >
> >All the tests that have been disabled are in:
> >https://issues.apache.org/jira/browse/HIVE-19509
> >I have created new issues to reenable each of them, they are linked
> to that issue.
> >Maybe I was slightly aggressive disabling some of the tests, however
> that seemed to be the only way to bring the tests failures with age count >
> 1 to zero.
> >
> >Instead of starting a vote to freeze the commits in another thread, I
> will start a vote to be stricter wrt committing to master, i.e., only
> commit if we get a clean QA run.
> >
> >We can discuss more about this issue over there.
> >
> >Thanks,
> >Jesús
> >
> >
> >
> >On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:
> >
> >Can we please make this freeze conditional, i.e. we unfreeze
> automatically
> >after ptest is clean (as evidenced by the clean HiveQA run on a
> given
> >JIRA).
> >
> >On 18/5/14, 15:16, "Alan Gates"  wrote:
> >
> >> We should do it in a separate thread so that people can see it with the
> >> [VOTE] subject.  Some people use that as a filter in their email to know
> >> when to pay attention to things.
> >>
> >> Alan.
> >>
> >> On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
> >> pjayachand...@hortonworks.com> wrote:
> >>
> >>> Will there be a separate voting thread? Or the voting on this thread is
> >>> sufficient for lock down?
> >>>
> >>> Thanks
> >>> Prasanth
> >>>
>  On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
> 
>  ​I see there's support for this, but people are still pouring in
> >>> commits.
>  I proposed we have a quick vote on this to lock down the commits
> >>> until we
>  get to green.  That way everyone knows we have drawn the line at a
> >>> specific
>  point.  Any commits after that point would be reverted.  There isn't a
>  category in the bylaws that fits this kind of vote but I suggest lazy
>  majority as the most appropriate one (at least 3 votes, more +1s than
>  -1s).
> 
>  Alan.​
> 
>  On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
> >>> vih...@cloudera.com>
>  wrote:
> 
> > I worked on a few quick-fix optimizations in Ptest infrastructure
> >>> over
> >>> the
> > weekend which reduced the execution run from ~90 min to ~70 min per
> >>> run. I
> > had to restart Ptest multiple times. I was resubmitting the patches
> >>> which
> > were in the queue manually, but I may have missed a few. In case you
> >>> have a
> > patch which is pending pre-commit and you don't see it in the queue,
> >>> please
> > submit it manually or let me know if you don't have access to the
> >>> jenkins
> > job. I will continue to work on the sub-tasks in HIVE-19425 and will
> >>> do
> > some maintenance next weekend as well.
> >
> > On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
> > jcama...@apache.org> wrote:
> >
> >> Vineet has already been working on disabling those tests that were
> >>> timing
> >> out. I am working on disabling those that are generating different q
> > 

Re: [VOTE] Stricter commit guidelines

[Siddharth Seth]

+1

On Mon, May 14, 2018 at 10:44 PM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

> After work has been done to ignore most of the tests that were failing
> consistently/intermittently [1], I wanted to start this vote to gather
> support from the community to be stricter wrt committing patches to Hive.
> The committers guide [2] already specifies that a +1 should be obtained
> before committing, but there is another clause that allows committing under
> the presence of flaky tests (clause 4). Flaky tests are as good as having
> no tests, hence I propose to remove clause 4 and enforce the +1 from
> testing infra before committing.
>
>
>
> As I see it, by enforcing that we always get a +1 from the testing infra
> before committing, 1) we will have a more stable project, and 2) we will
> have another incentive as a community to create a more robust testing
> infra, e.g., replacing flaky tests for similar unit tests that are not
> flaky, trying to decrease running time for tests, etc.
>
>
>
> Please, share your thoughts about this.
>
>
>
> Here is my +1.
>
>
>
> Thanks,
>
> Jesús
>
>
>
> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
>
> [2] https://cwiki.apache.org/confluence/display/Hive/
> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
>
>
>
>

[jira] [Created] (HIVE-19568) Active/Passive HS2 HA: Disallow direct connection to passive HS2 instance

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-19568:


 Summary: Active/Passive HS2 HA: Disallow direct connection to 
passive HS2 instance
 Key: HIVE-19568
 URL: https://issues.apache.org/jira/browse/HIVE-19568
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


The recommended usage for clients when connecting to HS2 with Active/Passive HA 
configuration is via ZK service discovery URL. But some applications do not 
support ZK service discovery in which case they use direct URL to connect to 
HS2 instance. If direct connection is to passive HS2 instance, the connection 
should be dropped with proper error message. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Publishing Hive RC artifacts

[Vineet Garg]

Hello,

I am unable to publish Hive 3.0 RC artifacts to maven. Following wiki page 
(https://cwiki.apache.org/confluence/display/Hive/HowToRelease#HowToRelease-Voting)
 I am running following command:


mvn deploy -DskipTests -Papache-release -Dmaven.javadoc.skip=true

But doing so result in error:

ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on 
project hive: Failed to deploy artifacts/metadata: Cannot access ${repourl} 
with type default using the available connector factories: 
BasicRepositoryConnectorFactory: Cannot access ${repourl} using the registered 
transporter factories: WagonTransporterFactory: Unsupported transport protocol 
-> [Help 1]
[ERROR]

Anybody know how do I fix this and publish RC artifacts to maven?

Vineet

[jira] [Created] (HIVE-19567) Fix flakiness in TestTriggers

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-19567:


 Summary: Fix flakiness in TestTriggers
 Key: HIVE-19567
 URL: https://issues.apache.org/jira/browse/HIVE-19567
 Project: Hive
  Issue Type: Bug
  Components: Test
Affects Versions: 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Identified another flakiness in TestTriggersMoveWorkloadManager which can cause 
intermittent test failures. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] Apache Hive 3.0.0 Release Candidate 0

[Vineet Garg]

Apache Hive 3.0.0 Release Candidate 0 is available here:

http://people.apache.org/~vgarg/apache-hive-3.0.0-rc-0


Tag: https://github.com/apache/hive/tree/release-3.0.0-rc0


Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Re: [DISCUSS] Unsustainable situation with ptests

[Prasanth Jayachandran]

Wow! Awesome. This is the 3rd time I remember seeing green run in >4yrs. :)

Thanks
Prasanth

> On May 15, 2018, at 5:28 PM, Jesus Camacho Rodriguez  
> wrote:
> 
> We have just had the first clean run in a while:
> https://builds.apache.org/job/PreCommit-HIVE-Build/10971/testReport/
> 
> I will continue monitoring follow-up runs.
> 
> Thanks,
> -Jesús
> 
> 
> On 5/14/18, 11:28 PM, "Prasanth Jayachandran" 
>  wrote:
> 
>Wondering if we can add a state transition from “Patch Available” to 
> “Ready To Commit” which can only be triggered by ptest bot on green test run.
> 
>Thanks
>Prasanth
> 
> 
> 
>On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" 
> > wrote:
> 
> 
>I have been working on fixing this situation while commits were still 
> coming in.
> 
>All the tests that have been disabled are in:
>https://issues.apache.org/jira/browse/HIVE-19509
>I have created new issues to reenable each of them, they are linked to 
> that issue.
>Maybe I was slightly aggressive disabling some of the tests, however that 
> seemed to be the only way to bring the tests failures with age count > 1 to 
> zero.
> 
>Instead of starting a vote to freeze the commits in another thread, I will 
> start a vote to be stricter wrt committing to master, i.e., only commit if we 
> get a clean QA run.
> 
>We can discuss more about this issue over there.
> 
>Thanks,
>Jesús
> 
> 
> 
>On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:
> 
>Can we please make this freeze conditional, i.e. we unfreeze 
> automatically
>after ptest is clean (as evidenced by the clean HiveQA run on a given
>JIRA).
> 
>On 18/5/14, 15:16, "Alan Gates"  wrote:
> 
>> We should do it in a separate thread so that people can see it with the
>> [VOTE] subject.  Some people use that as a filter in their email to know
>> when to pay attention to things.
>> 
>> Alan.
>> 
>> On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>> Will there be a separate voting thread? Or the voting on this thread is
>>> sufficient for lock down?
>>> 
>>> Thanks
>>> Prasanth
>>> 
 On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
 
 ​I see there's support for this, but people are still pouring in
>>> commits.
 I proposed we have a quick vote on this to lock down the commits
>>> until we
 get to green.  That way everyone knows we have drawn the line at a
>>> specific
 point.  Any commits after that point would be reverted.  There isn't a
 category in the bylaws that fits this kind of vote but I suggest lazy
 majority as the most appropriate one (at least 3 votes, more +1s than
 -1s).
 
 Alan.​
 
 On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
>>> vih...@cloudera.com>
 wrote:
 
> I worked on a few quick-fix optimizations in Ptest infrastructure
>>> over
>>> the
> weekend which reduced the execution run from ~90 min to ~70 min per
>>> run. I
> had to restart Ptest multiple times. I was resubmitting the patches
>>> which
> were in the queue manually, but I may have missed a few. In case you
>>> have a
> patch which is pending pre-commit and you don't see it in the queue,
>>> please
> submit it manually or let me know if you don't have access to the
>>> jenkins
> job. I will continue to work on the sub-tasks in HIVE-19425 and will
>>> do
> some maintenance next weekend as well.
> 
> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
> 
>> Vineet has already been working on disabling those tests that were
>>> timing
>> out. I am working on disabling those that are generating different q
> files
>> consistently for last ptests n runs. I am keeping track of all these
> tests
>> in https://issues.apache.org/jira/browse/HIVE-19509.
>> 
>> -Jesús
>> 
>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>   +1 on freezing commits until we get repetitive green tests. We
>>> should
>> probably disable (and remember in a jira to reenable then at later
>>> point)
>> tests that are flaky to get repetitive green test runs.
>> 
>>   Thanks
>>   Prasanth
>> 
>> 
>> 
>>   On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
> lirui.fu...@gmail.com
>>> wrote:
>> 
>> 
>>   +1 to freezing commits until we stabilize
>> 
>>   On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
>>   wrote:
>> 
>>> In order to understand the end-to-end precommit flow I would like
> to
>> get
>>> access to the PreCommit-HIVE-Build jenkins script. Does anyone one
>> know how
>>> can I get that?
>>> 
>>> On Fri, May 11, 2018 at 2:03 PM, Jesus 

Re: [DISCUSS] Unsustainable situation with ptests

[Jesus Camacho Rodriguez]

We have just had the first clean run in a while:
https://builds.apache.org/job/PreCommit-HIVE-Build/10971/testReport/

I will continue monitoring follow-up runs.

Thanks,
-Jesús


On 5/14/18, 11:28 PM, "Prasanth Jayachandran"  
wrote:

Wondering if we can add a state transition from “Patch Available” to “Ready 
To Commit” which can only be triggered by ptest bot on green test run.

Thanks
Prasanth



On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" 
> wrote:


I have been working on fixing this situation while commits were still 
coming in.

All the tests that have been disabled are in:
https://issues.apache.org/jira/browse/HIVE-19509
I have created new issues to reenable each of them, they are linked to that 
issue.
Maybe I was slightly aggressive disabling some of the tests, however that 
seemed to be the only way to bring the tests failures with age count > 1 to 
zero.

Instead of starting a vote to freeze the commits in another thread, I will 
start a vote to be stricter wrt committing to master, i.e., only commit if we 
get a clean QA run.

We can discuss more about this issue over there.

Thanks,
Jesús



On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:

Can we please make this freeze conditional, i.e. we unfreeze 
automatically
after ptest is clean (as evidenced by the clean HiveQA run on a given
JIRA).

On 18/5/14, 15:16, "Alan Gates"  wrote:

>We should do it in a separate thread so that people can see it with the
>[VOTE] subject.  Some people use that as a filter in their email to 
know
>when to pay attention to things.
>
>Alan.
>
>On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
>pjayachand...@hortonworks.com> wrote:
>
>> Will there be a separate voting thread? Or the voting on this thread 
is
>> sufficient for lock down?
>>
>> Thanks
>> Prasanth
>>
>> > On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
>> >
>> > ​I see there's support for this, but people are still pouring in
>>commits.
>> > I proposed we have a quick vote on this to lock down the commits
>>until we
>> > get to green.  That way everyone knows we have drawn the line at a
>> specific
>> > point.  Any commits after that point would be reverted.  There 
isn't a
>> > category in the bylaws that fits this kind of vote but I suggest 
lazy
>> > majority as the most appropriate one (at least 3 votes, more +1s 
than
>> > -1s).
>> >
>> > Alan.​
>> >
>> > On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
>> vih...@cloudera.com>
>> > wrote:
>> >
>> >> I worked on a few quick-fix optimizations in Ptest infrastructure
>>over
>> the
>> >> weekend which reduced the execution run from ~90 min to ~70 min 
per
>> run. I
>> >> had to restart Ptest multiple times. I was resubmitting the 
patches
>> which
>> >> were in the queue manually, but I may have missed a few. In case 
you
>> have a
>> >> patch which is pending pre-commit and you don't see it in the 
queue,
>> please
>> >> submit it manually or let me know if you don't have access to the
>> jenkins
>> >> job. I will continue to work on the sub-tasks in HIVE-19425 and 
will
>>do
>> >> some maintenance next weekend as well.
>> >>
>> >> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
>> >> jcama...@apache.org> wrote:
>> >>
>> >>> Vineet has already been working on disabling those tests that 
were
>> timing
>> >>> out. I am working on disabling those that are generating 
different q
>> >> files
>> >>> consistently for last ptests n runs. I am keeping track of all 
these
>> >> tests
>> >>> in https://issues.apache.org/jira/browse/HIVE-19509.
>> >>>
>> >>> -Jesús
>> >>>
>> >>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
>> >>> pjayachand...@hortonworks.com> wrote:
>> >>>
>> >>>+1 on freezing commits until we get repetitive green tests. We
>> should
>> >>> probably disable (and remember in a jira to reenable then at 
later
>> point)
>> >>> tests that are flaky to get repetitive green test runs.
>> >>>
>> >>>Thanks
>> >>>Prasanth
>> >>>
>> >>>
>> >>>
>> >>>On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
>> >> lirui.fu...@gmail.com
>> >>> > wrote:
>> >>>
>> >>>
   

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

[Andrew Sherman via Review Board]

> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
> looks like it does already
> 
> Sahil Takiar wrote:
> then is the tez specific code necessary?
> 
> Andrew Sherman wrote:
> I think it is, as tez has its own hive-site.xml but I am not 100% sure
> 
> Sahil Takiar wrote:
> yes, tez has its own `hive-site.xml`, as does Spark; so do we need to do 
> this for Spark too?

That would make sense but then we have to decide if that is 
data/conf/spark/local or data/conf/spark/local :-(
These code paths are confusing becaus it is hard to anticipate how they'll be 
used.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67073: HIVE-19370 : Retain time part in add_months function on timestamp datatype fields in hive

[Bharathkrishna Guruvayoor Murali via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67073/
---

(Updated May 15, 2018, 11:18 p.m.)


Review request for hive, Peter Vary, Sahil Takiar, and Vihang Karajgaonkar.


Changes
---

Changing the patch with new logic.
Adding an additional optional parameter dateFormat, which can provide output 
format that user expects.
The default format is kept as the existing one, -MM-dd. 
Hence, the patch is backward compatible, and also capable of handling different 
input and output formats as required by the user.

Also, now the code format is consistent with the other similar UDFs 
MonthsBetween and GenericUDFDateFormat.

Added unit tests to check various cases.

Thanks Peter for helping me reach a conclusion on what logic I need to use.

I have updated the UDF description. Will update the docs once the patch is 
approved.


Repository: hive-git


Description
---

Adding support to retain the time part (HH:mm:ss) for add_months UDF when the 
input is given as timestamp format.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAddMonths.java 
dae4b97b4a17e98122431e5fda655fd9f873fdb5 
  
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFAddMonths.java 
af9b6c43c7dafc69c4944eab02894786af306f35 


Diff: https://reviews.apache.org/r/67073/diff/2/

Changes: https://reviews.apache.org/r/67073/diff/1-2/


Testing
---

Added unit tests.


Thanks,

Bharathkrishna Guruvayoor Murali

Re: [VOTE] Stricter commit guidelines

[Vihang Karajgaonkar]

+1 Default retry on failing test should help. Another way to identify test
issues is to run them in the same batch as being run in the precommit job.

On Tue, May 15, 2018 at 4:11 PM, Deepak Jaiswal 
wrote:

> +1
>
> On 5/15/18, 4:06 PM, "Jesus Camacho Rodriguez" 
> wrote:
>
> That is already part of the policy although it apparently remained
> unnoticed:
>
> *If a commit introduces new test failures, the preferred process is to
> revert the patch, rather than opening a new JIRA to fix the new failures.*
>
> It can be reworded to be stricter... But in any case, we should all
> enforce it from now on.
>
> -Jesús
>
>
> On 5/15/18, 3:55 PM, "Sergey Shelukhin" 
> wrote:
>
> +1. Can we also add something about revert-first policy if some
> patch
> breaks tests?
> So that it’s ok to revert if the tests aren’t fixed quickly with a
> follow-up.
>
> On 18/5/15, 13:08, "Jesus Camacho Rodriguez" 
> wrote:
>
> >I was hoping that by being stricter, we are going to make an
> effort to
> >fix the flaky
> >tests. The reason is precisely what you mention: if you cannot
> commit,
> >you need
> >to fix this situation. That is what I meant with providing an
> additional
> >incentive to
> >make testing more robust: currently there is no incentive. I do
> not think
> >that
> >improving tests is a responsibility of a single developer, but
> rather a
> >responsibility
> >of all of us. Disabling the tests is a one-time solution to get
> to a
> >clean run, trying to
> >accelerate the process to get to it, as we did not want to block
> >development for
> >weeks. Then flaky tests should just not go in, and if they do, we
> can
> >just revert
> >the patch (this is what [2] says btw).
> >
> >The first thing we need to do is identifying why a test is flaky.
> After
> >examining runs
> >for the last few days, I saw many of them fall in following
> categories:
> >- Many of them are flaky because of estimations such as data
> size. One
> >possible
> >solution is to mask data size for those tests, as we already mask
> some
> >environment
> >dependent information.
> >- Some of them are flaky because environment issues, e.g., I see
> this a
> >lot with
> >TestTriggersMoveWorkloadManager. If their logic cannot be
> rewritten, a
> >possible
> >solution is to add a max number of retries selectively for those
> tests
> >(surefire has
> >a rerunFailingTestsCount option that I am not familiar with),
> expecting
> >that they
> >pass at least once.
> >
> >Not sure if you have other ideas?
> >
> >-Jesús
> >
> >
> >On 5/15/18, 11:59 AM, "Vihang Karajgaonkar" 
> wrote:
> >
> >Can we also define a standard process to identify a flaky
> test and
> >thereby
> >making it eligible to be disabled? I am worried that the
> intermittent
> >the
> >flaky ones will stall the patches when we restart allowing the
> >commits.
> >
> >On Tue, May 15, 2018 at 10:50 AM, Vineet Garg <
> vg...@hortonworks.com>
> >wrote:
> >
> >> +1
> >>
> >> > On May 15, 2018, at 9:13 AM, Alan Gates <
> alanfga...@gmail.com>
> >wrote:
> >> >
> >> > +1.
> >> >
> >> > Alan.
> >> >
> >> > On Tue, May 15, 2018 at 9:12 AM, Sergio Pena
> >
> >> > wrote:
> >> >
> >> >> +1
> >> >>
> >> >> On Tue, May 15, 2018 at 11:05 AM, Gunther Hagleitner <
> >> >> ghagleit...@hortonworks.com> wrote:
> >> >>
> >> >>> +1
> >> >>> 
> >> >>> From: Sankar Hariappan 
> >> >>> Sent: Tuesday, May 15, 2018 9:03 AM
> >> >>> To: dev@hive.apache.org
> >> >>> Subject: Re: [VOTE] Stricter commit guidelines
> >> >>>
> >> >>> +1
> >> >>>
> >> >>>
> >> >>> On 15/05/18, 9:30 PM, "Sahil Takiar" <
> takiar.sa...@gmail.com>
> >wrote:
> >> >>>
> >>  +1
> >> 
> >>  On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley <
> >> owen.omal...@gmail.com
> >> >>>
> >>  wrote:
> >> 
> >> > +1
> >> >
> >> > On Tue, May 15, 2018 at 8:55 AM, Peter Vary
> 

Re: [VOTE] Stricter commit guidelines

[Deepak Jaiswal]

+1

On 5/15/18, 4:06 PM, "Jesus Camacho Rodriguez"  wrote:

That is already part of the policy although it apparently remained 
unnoticed:

*If a commit introduces new test failures, the preferred process is to 
revert the patch, rather than opening a new JIRA to fix the new failures.*

It can be reworded to be stricter... But in any case, we should all enforce 
it from now on.

-Jesús


On 5/15/18, 3:55 PM, "Sergey Shelukhin"  wrote:

+1. Can we also add something about revert-first policy if some patch
breaks tests?
So that it’s ok to revert if the tests aren’t fixed quickly with a
follow-up.

On 18/5/15, 13:08, "Jesus Camacho Rodriguez"  
wrote:

>I was hoping that by being stricter, we are going to make an effort to
>fix the flaky
>tests. The reason is precisely what you mention: if you cannot commit,
>you need
>to fix this situation. That is what I meant with providing an 
additional
>incentive to
>make testing more robust: currently there is no incentive. I do not 
think
>that
>improving tests is a responsibility of a single developer, but rather a
>responsibility
>of all of us. Disabling the tests is a one-time solution to get to a
>clean run, trying to
>accelerate the process to get to it, as we did not want to block
>development for
>weeks. Then flaky tests should just not go in, and if they do, we can
>just revert
>the patch (this is what [2] says btw).
>
>The first thing we need to do is identifying why a test is flaky. After
>examining runs
>for the last few days, I saw many of them fall in following categories:
>- Many of them are flaky because of estimations such as data size. One
>possible
>solution is to mask data size for those tests, as we already mask some
>environment
>dependent information.
>- Some of them are flaky because environment issues, e.g., I see this a
>lot with
>TestTriggersMoveWorkloadManager. If their logic cannot be rewritten, a
>possible
>solution is to add a max number of retries selectively for those tests
>(surefire has
>a rerunFailingTestsCount option that I am not familiar with), expecting
>that they
>pass at least once.
>
>Not sure if you have other ideas?
>
>-Jesús
>
>
>On 5/15/18, 11:59 AM, "Vihang Karajgaonkar"  
wrote:
>
>Can we also define a standard process to identify a flaky test and
>thereby
>making it eligible to be disabled? I am worried that the 
intermittent
>the
>flaky ones will stall the patches when we restart allowing the
>commits.
>
>On Tue, May 15, 2018 at 10:50 AM, Vineet Garg 

>wrote:
>
>> +1
>>
>> > On May 15, 2018, at 9:13 AM, Alan Gates 
>wrote:
>> >
>> > +1.
>> >
>> > Alan.
>> >
>> > On Tue, May 15, 2018 at 9:12 AM, Sergio Pena
>
>> > wrote:
>> >
>> >> +1
>> >>
>> >> On Tue, May 15, 2018 at 11:05 AM, Gunther Hagleitner <
>> >> ghagleit...@hortonworks.com> wrote:
>> >>
>> >>> +1
>> >>> 
>> >>> From: Sankar Hariappan 
>> >>> Sent: Tuesday, May 15, 2018 9:03 AM
>> >>> To: dev@hive.apache.org
>> >>> Subject: Re: [VOTE] Stricter commit guidelines
>> >>>
>> >>> +1
>> >>>
>> >>>
>> >>> On 15/05/18, 9:30 PM, "Sahil Takiar" 
>wrote:
>> >>>
>>  +1
>> 
>>  On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley <
>> owen.omal...@gmail.com
>> >>>
>>  wrote:
>> 
>> > +1
>> >
>> > On Tue, May 15, 2018 at 8:55 AM, Peter Vary
>
>> >> wrote:
>> >
>> >> +1 - Hoping for something like this for a long while! 
Thanks
>for
>> >>> taking
>> >> this up all!
>> >>
>> >>> On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
>> >> jcama...@apache.org> wrote:
>> >>>
>> >>> Forgot to mention the length 

Re: [VOTE] Stricter commit guidelines

[Jesus Camacho Rodriguez]

That is already part of the policy although it apparently remained unnoticed:

*If a commit introduces new test failures, the preferred process is to revert 
the patch, rather than opening a new JIRA to fix the new failures.*

It can be reworded to be stricter... But in any case, we should all enforce it 
from now on.

-Jesús


On 5/15/18, 3:55 PM, "Sergey Shelukhin"  wrote:

+1. Can we also add something about revert-first policy if some patch
breaks tests?
So that it’s ok to revert if the tests aren’t fixed quickly with a
follow-up.

On 18/5/15, 13:08, "Jesus Camacho Rodriguez"  wrote:

>I was hoping that by being stricter, we are going to make an effort to
>fix the flaky
>tests. The reason is precisely what you mention: if you cannot commit,
>you need
>to fix this situation. That is what I meant with providing an additional
>incentive to
>make testing more robust: currently there is no incentive. I do not think
>that
>improving tests is a responsibility of a single developer, but rather a
>responsibility
>of all of us. Disabling the tests is a one-time solution to get to a
>clean run, trying to
>accelerate the process to get to it, as we did not want to block
>development for
>weeks. Then flaky tests should just not go in, and if they do, we can
>just revert
>the patch (this is what [2] says btw).
>
>The first thing we need to do is identifying why a test is flaky. After
>examining runs
>for the last few days, I saw many of them fall in following categories:
>- Many of them are flaky because of estimations such as data size. One
>possible
>solution is to mask data size for those tests, as we already mask some
>environment
>dependent information.
>- Some of them are flaky because environment issues, e.g., I see this a
>lot with
>TestTriggersMoveWorkloadManager. If their logic cannot be rewritten, a
>possible
>solution is to add a max number of retries selectively for those tests
>(surefire has
>a rerunFailingTestsCount option that I am not familiar with), expecting
>that they
>pass at least once.
>
>Not sure if you have other ideas?
>
>-Jesús
>
>
>On 5/15/18, 11:59 AM, "Vihang Karajgaonkar"  wrote:
>
>Can we also define a standard process to identify a flaky test and
>thereby
>making it eligible to be disabled? I am worried that the intermittent
>the
>flaky ones will stall the patches when we restart allowing the
>commits.
>
>On Tue, May 15, 2018 at 10:50 AM, Vineet Garg 
>wrote:
>
>> +1
>>
>> > On May 15, 2018, at 9:13 AM, Alan Gates 
>wrote:
>> >
>> > +1.
>> >
>> > Alan.
>> >
>> > On Tue, May 15, 2018 at 9:12 AM, Sergio Pena
>
>> > wrote:
>> >
>> >> +1
>> >>
>> >> On Tue, May 15, 2018 at 11:05 AM, Gunther Hagleitner <
>> >> ghagleit...@hortonworks.com> wrote:
>> >>
>> >>> +1
>> >>> 
>> >>> From: Sankar Hariappan 
>> >>> Sent: Tuesday, May 15, 2018 9:03 AM
>> >>> To: dev@hive.apache.org
>> >>> Subject: Re: [VOTE] Stricter commit guidelines
>> >>>
>> >>> +1
>> >>>
>> >>>
>> >>> On 15/05/18, 9:30 PM, "Sahil Takiar" 
>wrote:
>> >>>
>>  +1
>> 
>>  On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley <
>> owen.omal...@gmail.com
>> >>>
>>  wrote:
>> 
>> > +1
>> >
>> > On Tue, May 15, 2018 at 8:55 AM, Peter Vary
>
>> >> wrote:
>> >
>> >> +1 - Hoping for something like this for a long while! Thanks
>for
>> >>> taking
>> >> this up all!
>> >>
>> >>> On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
>> >> jcama...@apache.org> wrote:
>> >>>
>> >>> Forgot to mention the length of the vote in original
>message.
>> >>>
>> >>> Let's leave the vote open for a shorter period than usual,
>for
>> >>> instance
>> >> 48 hours, i.e., till Wednesday 10pm PST. Situation can only
>get
>> >> worse
>> > than
>> >> it is now if we do not take action for a longer period.
>> >>>
>> >>> As Alan suggested, vote passes if there is a lazy majority
>(at
>> >>> least 3
>> >> votes, more +1s than -1s).
>> >>>
>> >>> 

Re: [VOTE] Stricter commit guidelines

[Sergey Shelukhin]

+1. Can we also add something about revert-first policy if some patch
breaks tests?
So that it’s ok to revert if the tests aren’t fixed quickly with a
follow-up.

On 18/5/15, 13:08, "Jesus Camacho Rodriguez"  wrote:

>I was hoping that by being stricter, we are going to make an effort to
>fix the flaky
>tests. The reason is precisely what you mention: if you cannot commit,
>you need
>to fix this situation. That is what I meant with providing an additional
>incentive to
>make testing more robust: currently there is no incentive. I do not think
>that
>improving tests is a responsibility of a single developer, but rather a
>responsibility
>of all of us. Disabling the tests is a one-time solution to get to a
>clean run, trying to
>accelerate the process to get to it, as we did not want to block
>development for
>weeks. Then flaky tests should just not go in, and if they do, we can
>just revert
>the patch (this is what [2] says btw).
>
>The first thing we need to do is identifying why a test is flaky. After
>examining runs
>for the last few days, I saw many of them fall in following categories:
>- Many of them are flaky because of estimations such as data size. One
>possible
>solution is to mask data size for those tests, as we already mask some
>environment
>dependent information.
>- Some of them are flaky because environment issues, e.g., I see this a
>lot with
>TestTriggersMoveWorkloadManager. If their logic cannot be rewritten, a
>possible
>solution is to add a max number of retries selectively for those tests
>(surefire has
>a rerunFailingTestsCount option that I am not familiar with), expecting
>that they
>pass at least once.
>
>Not sure if you have other ideas?
>
>-Jesús
>
>
>On 5/15/18, 11:59 AM, "Vihang Karajgaonkar"  wrote:
>
>Can we also define a standard process to identify a flaky test and
>thereby
>making it eligible to be disabled? I am worried that the intermittent
>the
>flaky ones will stall the patches when we restart allowing the
>commits.
>
>On Tue, May 15, 2018 at 10:50 AM, Vineet Garg 
>wrote:
>
>> +1
>>
>> > On May 15, 2018, at 9:13 AM, Alan Gates 
>wrote:
>> >
>> > +1.
>> >
>> > Alan.
>> >
>> > On Tue, May 15, 2018 at 9:12 AM, Sergio Pena
>
>> > wrote:
>> >
>> >> +1
>> >>
>> >> On Tue, May 15, 2018 at 11:05 AM, Gunther Hagleitner <
>> >> ghagleit...@hortonworks.com> wrote:
>> >>
>> >>> +1
>> >>> 
>> >>> From: Sankar Hariappan 
>> >>> Sent: Tuesday, May 15, 2018 9:03 AM
>> >>> To: dev@hive.apache.org
>> >>> Subject: Re: [VOTE] Stricter commit guidelines
>> >>>
>> >>> +1
>> >>>
>> >>>
>> >>> On 15/05/18, 9:30 PM, "Sahil Takiar" 
>wrote:
>> >>>
>>  +1
>> 
>>  On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley <
>> owen.omal...@gmail.com
>> >>>
>>  wrote:
>> 
>> > +1
>> >
>> > On Tue, May 15, 2018 at 8:55 AM, Peter Vary
>
>> >> wrote:
>> >
>> >> +1 - Hoping for something like this for a long while! Thanks
>for
>> >>> taking
>> >> this up all!
>> >>
>> >>> On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
>> >> jcama...@apache.org> wrote:
>> >>>
>> >>> Forgot to mention the length of the vote in original
>message.
>> >>>
>> >>> Let's leave the vote open for a shorter period than usual,
>for
>> >>> instance
>> >> 48 hours, i.e., till Wednesday 10pm PST. Situation can only
>get
>> >> worse
>> > than
>> >> it is now if we do not take action for a longer period.
>> >>>
>> >>> As Alan suggested, vote passes if there is a lazy majority
>(at
>> >>> least 3
>> >> votes, more +1s than -1s).
>> >>>
>> >>> Thanks,
>> >>> Jesús
>> >>>
>> >>>
>> >>> On 5/15/18, 8:37 AM, "Andrew Sherman"
>
>> >>> wrote:
>> >>>
>> >>>   +1
>> >>>
>> >>>   On Tue, May 15, 2018 at 2:34 AM Rui Li
>
>> > wrote:
>> >>>
>>  +1
>> 
>>  On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
>>  pjayachand...@hortonworks.com> wrote:
>> 
>> > +1
>> >
>> >
>> >
>> > Thanks
>> > Prasanth
>> >
>> >
>> >
>> > On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho
>> >> Rodriguez"
>> >>> <
>> > jcama...@apache.org> wrote:
>> >
>> >
>> > After work has been done 

[jira] [Created] (HIVE-19566) Vectorization: Fix NULL / Wrong Results issues in Complex Type Functions

[Matt McCline (JIRA)]

Matt McCline created HIVE-19566:
---

 Summary: Vectorization: Fix NULL / Wrong Results issues in Complex 
Type Functions
 Key: HIVE-19566
 URL: https://issues.apache.org/jira/browse/HIVE-19566
 Project: Hive
  Issue Type: Bug
Reporter: Matt McCline
 Fix For: 3.1.0


Write new UT tests that use random data and intentional isRepeating batches to 
checks for NULL and Wrong Results for vectorized Complex Type functions:
 * index
 * (StructField)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19565) Vectorization: Fix NULL / Wrong Results issues in STRING Functions

[Matt McCline (JIRA)]

Matt McCline created HIVE-19565:
---

 Summary: Vectorization: Fix NULL / Wrong Results issues in STRING 
Functions
 Key: HIVE-19565
 URL: https://issues.apache.org/jira/browse/HIVE-19565
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline


Write new UT tests that use random data and intentional isRepeating batches to 
checks for NULL and Wrong Results for vectorized STRING functions:
 * char_length
 * concat
 * initcap
 * length
 * lower
 * ltrim
 * octet_length
 * regexp
 * rtrim
 * trim
 * upper
 * UDF:
 ** hex
 ** like
 ** substr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

[Sahil Takiar]

> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
> looks like it does already
> 
> Sahil Takiar wrote:
> then is the tez specific code necessary?
> 
> Andrew Sherman wrote:
> I think it is, as tez has its own hive-site.xml but I am not 100% sure

yes, tez has its own `hive-site.xml`, as does Spark; so do we need to do this 
for Spark too?


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

[jira] [Created] (HIVE-19564) Vectorization: Fix NULL / Wrong Results issues in Functions

[Matt McCline (JIRA)]

Matt McCline created HIVE-19564:
---

 Summary: Vectorization: Fix NULL / Wrong Results issues in 
Functions
 Key: HIVE-19564
 URL: https://issues.apache.org/jira/browse/HIVE-19564
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline


Write new UT tests that use random data and intentional isRepeating batches to 
checks for NULL and Wrong Results for vectorized functions:
 * Generic UDF Functions
 ** abs
 ** bround
 ** ceiling
 ** floor
 ** pmod
 ** power
 ** round
 * UDF Functions
 ** Acos
 ** Asin
 ** Atan
 ** Bin
 ** Cos
 ** Degrees
 ** Exp
 ** Ln
 ** Log
 ** log10
 ** log2
 ** radians
 ** rand
 ** sign
 ** sin
 ** sqrt
 ** tan



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

[Sahil Takiar]

> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > 
> >
> > why would we want the ec commands to work outside the 
> > `TestErasureCodingHDFSCliDriver`?
> 
> Andrew Sherman wrote:
> So you could run an ec test in TestCliDriver (sorry reviewboard lost my 
> ealrier reply)
> 
> Sahil Takiar wrote:
> why would you want to do that? whats the use case? shouldn't 
> `TestErasureCodingHDFSCliDriver` encapsulate all EC-related tests? plus I'm 
> not sure how you could run any EC commands against a local filesystem, 
> wouldn't they all be no-ops?
> 
> Andrew Sherman wrote:
> They would. When I'm developing an EC test I may sometimes want to run 
> the test in TestCLiDriver with erasure commands being no-ops as a way to 
> validate the script. Myabe this is weird, but I've been doing it.

I see, makes sense.


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 66862: HIVE-19258 add originals support to MM tables (and make the conversion a metadata only operation)

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66862/
---

(Updated May 15, 2018, 9:34 p.m.)


Review request for hive and Thejas Nair.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0a997a1569 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 de61d717fc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 63fe8adc8b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 969c591917 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 183515a0ed 
  ql/src/java/org/apache/hadoop/hive/ql/io/BucketizedHiveInputFormat.java 
75fa09de8e 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3d965c0515 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 2337a350e6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/io/FileOperations.java 
b3e76b6259 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 
80f77b9f0c 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
b698c84080 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 3e2784ba2d 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java a88a570c52 
  ql/src/test/queries/clientpositive/mm_conversions.q 55565a9428 
  ql/src/test/results/clientpositive/llap/mm_conversions.q.out 4754710291 


Diff: https://reviews.apache.org/r/66862/diff/7/

Changes: https://reviews.apache.org/r/66862/diff/6-7/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-19563) Flaky test: TestMiniLlapLocalCliDriver.tez_vector_dynpart_hashjoin_1

[Jason Dere (JIRA)]

Jason Dere created HIVE-19563:
-

 Summary: Flaky test: 
TestMiniLlapLocalCliDriver.tez_vector_dynpart_hashjoin_1
 Key: HIVE-19563
 URL: https://issues.apache.org/jira/browse/HIVE-19563
 Project: Hive
  Issue Type: Sub-task
  Components: Tests
Reporter: Jason Dere


{noformat}
Client Execution succeeded but contained differences (error code = 1) after 
executing tez_vector_dynpart_hashjoin_1.q 
407c407
< -13036 1
---
> -8915 1
410c410
< -8915 1
---
> -13036 1
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67102: HIVE-19440: Make StorageBasedAuthorizer work with information schema

[Daniel Dai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67102/
---

(Updated May 15, 2018, 9:04 p.m.)


Review request for hive.


Repository: hive-git


Description
---

See HIVE-19440


Diffs (updated)
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/storagehandler/DummyHCatAuthProvider.java
 a53028f 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestHDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
itests/util/src/main/java/org/apache/hadoop/hive/ql/security/DummyHiveMetastoreAuthorizationProvider.java
 31e795c 
  metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql d9606d8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java a1f549a 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HDFSPermissionPolicyProvider.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProviderBase.java
 8a7c06d 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveMetastoreAuthorizationProvider.java
 0dab334 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PolicyProviderContainer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/PrivilegeSynchonizer.java
 9b2e6cd 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java
 b66d188 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/HiveV1Authorizer.java
 48798d8 
  
ql/src/java/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/SQLAuthorizationUtils.java
 02ed7aa 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentAuthorizer.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
 3eb0914 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 661beb5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 92d2e3f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 6af2aa5 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 09f9bb1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 264fdb9 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 ce7d286 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 b223920 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/client/builder/HiveObjectPrivilegeBuilder.java
 d802e1a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MDBPrivilege.java
 3d8fa21 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MGlobalPrivilege.java
 5b496e0 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionColumnPrivilege.java
 ab50a92 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MPartitionPrivilege.java
 3193bc1 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTableColumnPrivilege.java
 ad7322f 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/model/MTablePrivilege.java
 6460400 
  standalone-metastore/src/main/resources/package.jdo 2d2cb19 
  standalone-metastore/src/main/sql/derby/hive-schema-3.1.0.derby.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/derby/upgrade-3.0.0-to-3.1.0.derby.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mssql/hive-schema-3.1.0.mssql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mssql/upgrade-3.0.0-to-3.1.0.mssql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mysql/hive-schema-3.1.0.mysql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/mysql/upgrade-3.0.0-to-3.1.0.mysql.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/oracle/hive-schema-3.1.0.oracle.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/oracle/upgrade-3.0.0-to-3.1.0.oracle.sql 
PRE-CREATION 
  standalone-metastore/src/main/sql/postgres/hive-schema-3.1.0.postgres.sql 
PRE-CREATION 
  
standalone-metastore/src/main/sql/postgres/upgrade-3.0.0-to-3.1.0.postgres.sql 
PRE-CREATION 
  standalone-metastore/src/main/thrift/hive_metastore.thrift 19d4433 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f6899be 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 98a85cc 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 7186add 


Diff: https://reviews.apache.org/r/67102/diff/2/

Changes: https://reviews.apache.org/r/67102/diff/1-2/


Testing
---


Thanks,

Daniel Dai

[jira] [Created] (HIVE-19562) Flaky test: TestMiniSparkOnYarn FileNotFoundException in spark-submit

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-19562:
---

 Summary: Flaky test: TestMiniSparkOnYarn FileNotFoundException in 
spark-submit
 Key: HIVE-19562
 URL: https://issues.apache.org/jira/browse/HIVE-19562
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar


Seeing sporadic failures during test setup. Specifically, when spark-submit 
runs this error (or a similar error) gets thrown:

{code}
2018-05-15T10:55:02,112  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient: Exception in thread "main" 
java.io.FileNotFoundException: File 
file:/tmp/spark-56e217f7-b8a5-4c63-9a6b-d737a64f2820/__spark_libs__7371510645900072447.zip
 does not exist
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:356)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:478)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:565)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:863)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.yarn.Client.run(Client.scala:1146)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1518)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
2018-05-15T10:55:02,113  INFO 
[RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main] 
client.SparkSubmitSparkClient:  at 
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}

Essentially, Spark is writing some files for container localization to a tmp 
dir, and that tmp dir is getting deleted. We have seen a lot of issues with 
writing files to {{/tmp/}} in the past, so 

[jira] [Created] (HIVE-19561) Update README.md to update requirements for Hadoop

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-19561:
--

 Summary: Update README.md to update requirements for Hadoop
 Key: HIVE-19561
 URL: https://issues.apache.org/jira/browse/HIVE-19561
 Project: Hive
  Issue Type: Task
Reporter: Vineet Garg
Assignee: Vineet Garg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 67140: HIVE-19560: Retry test runner and retry rule for flaky tests

[j . prasanth . j]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67140/
---

Review request for hive and Jesús Camacho Rodríguez.


Bugs: HIVE-19560
https://issues.apache.org/jira/browse/HIVE-19560


Repository: hive-git


Description
---

HIVE-19560: Retry test runner and retry rule for flaky tests


Diffs
-

  common/src/test/org/apache/hive/common/util/Retry.java PRE-CREATION 
  common/src/test/org/apache/hive/common/util/RetryTestRunner.java PRE-CREATION 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 7e17d5d888755c7fb19bc34a712db5bd6fee4d32 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestTriggersMoveWorkloadManager.java
 e017e6382f752580cb7e2d1704fab34ad9ac001e 


Diff: https://reviews.apache.org/r/67140/diff/1/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-19560) Retry test runner and retry rule for flaky tests

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-19560:


 Summary: Retry test runner and retry rule for flaky tests
 Key: HIVE-19560
 URL: https://issues.apache.org/jira/browse/HIVE-19560
 Project: Hive
  Issue Type: Improvement
  Components: Test
Affects Versions: 3.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Implement custom test runner that retries failed tests as a workaround for 
flakiness. Also a test rule for retrying failed tests (for cases where custom 
test runner is not possible, ParametrizedTests already uses a TestRunner). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19559) SparkClientImpl shouldn't name redirector thread "RemoteDriver"

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-19559:
---

 Summary: SparkClientImpl shouldn't name redirector thread 
"RemoteDriver"
 Key: HIVE-19559
 URL: https://issues.apache.org/jira/browse/HIVE-19559
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Sahil Takiar


The thread responsible for re-directing the stdout / stderr of the spark-submit 
process are named {{RemoteDriver...}} which is misleading because these threads 
are not re-directing output from the {{RemoteDriver}}, just from the 
spark-submit stdout / stderr.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Stricter commit guidelines

[Jesus Camacho Rodriguez]

I was hoping that by being stricter, we are going to make an effort to fix the 
flaky
tests. The reason is precisely what you mention: if you cannot commit, you need
to fix this situation. That is what I meant with providing an additional 
incentive to
make testing more robust: currently there is no incentive. I do not think that
improving tests is a responsibility of a single developer, but rather a 
responsibility
of all of us. Disabling the tests is a one-time solution to get to a clean run, 
trying to
accelerate the process to get to it, as we did not want to block development for
weeks. Then flaky tests should just not go in, and if they do, we can just 
revert
the patch (this is what [2] says btw).

The first thing we need to do is identifying why a test is flaky. After 
examining runs
for the last few days, I saw many of them fall in following categories:
- Many of them are flaky because of estimations such as data size. One possible
solution is to mask data size for those tests, as we already mask some 
environment
dependent information.
- Some of them are flaky because environment issues, e.g., I see this a lot with
TestTriggersMoveWorkloadManager. If their logic cannot be rewritten, a possible
solution is to add a max number of retries selectively for those tests 
(surefire has
a rerunFailingTestsCount option that I am not familiar with), expecting that 
they
pass at least once.

Not sure if you have other ideas?

-Jesús


On 5/15/18, 11:59 AM, "Vihang Karajgaonkar"  wrote:

Can we also define a standard process to identify a flaky test and thereby
making it eligible to be disabled? I am worried that the intermittent the
flaky ones will stall the patches when we restart allowing the commits.

On Tue, May 15, 2018 at 10:50 AM, Vineet Garg  wrote:

> +1
>
> > On May 15, 2018, at 9:13 AM, Alan Gates  wrote:
> >
> > +1.
> >
> > Alan.
> >
> > On Tue, May 15, 2018 at 9:12 AM, Sergio Pena 
> > wrote:
> >
> >> +1
> >>
> >> On Tue, May 15, 2018 at 11:05 AM, Gunther Hagleitner <
> >> ghagleit...@hortonworks.com> wrote:
> >>
> >>> +1
> >>> 
> >>> From: Sankar Hariappan 
> >>> Sent: Tuesday, May 15, 2018 9:03 AM
> >>> To: dev@hive.apache.org
> >>> Subject: Re: [VOTE] Stricter commit guidelines
> >>>
> >>> +1
> >>>
> >>>
> >>> On 15/05/18, 9:30 PM, "Sahil Takiar"  wrote:
> >>>
>  +1
> 
>  On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley <
> owen.omal...@gmail.com
> >>>
>  wrote:
> 
> > +1
> >
> > On Tue, May 15, 2018 at 8:55 AM, Peter Vary 
> >> wrote:
> >
> >> +1 - Hoping for something like this for a long while! Thanks for
> >>> taking
> >> this up all!
> >>
> >>> On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
> >> jcama...@apache.org> wrote:
> >>>
> >>> Forgot to mention the length of the vote in original message.
> >>>
> >>> Let's leave the vote open for a shorter period than usual, for
> >>> instance
> >> 48 hours, i.e., till Wednesday 10pm PST. Situation can only get
> >> worse
> > than
> >> it is now if we do not take action for a longer period.
> >>>
> >>> As Alan suggested, vote passes if there is a lazy majority (at
> >>> least 3
> >> votes, more +1s than -1s).
> >>>
> >>> Thanks,
> >>> Jesús
> >>>
> >>>
> >>> On 5/15/18, 8:37 AM, "Andrew Sherman" 
> >>> wrote:
> >>>
> >>>   +1
> >>>
> >>>   On Tue, May 15, 2018 at 2:34 AM Rui Li 
> > wrote:
> >>>
>  +1
> 
>  On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
>  pjayachand...@hortonworks.com> wrote:
> 
> > +1
> >
> >
> >
> > Thanks
> > Prasanth
> >
> >
> >
> > On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho
> >> Rodriguez"
> >>> <
> > jcama...@apache.org> wrote:
> >
> >
> > After work has been done to ignore most of the tests that were
> > failing
> > consistently/intermittently [1], I wanted to start this vote to
> > gather
> > support from the community to be stricter wrt committing patches
> >>> to
> >> Hive.
> > The committers guide [2] already specifies that a +1 should be
> > obtained
> > 

Re: [VOTE] Stricter commit guidelines

[Vihang Karajgaonkar]

Can we also define a standard process to identify a flaky test and thereby
making it eligible to be disabled? I am worried that the intermittent the
flaky ones will stall the patches when we restart allowing the commits.

On Tue, May 15, 2018 at 10:50 AM, Vineet Garg  wrote:

> +1
>
> > On May 15, 2018, at 9:13 AM, Alan Gates  wrote:
> >
> > +1.
> >
> > Alan.
> >
> > On Tue, May 15, 2018 at 9:12 AM, Sergio Pena 
> > wrote:
> >
> >> +1
> >>
> >> On Tue, May 15, 2018 at 11:05 AM, Gunther Hagleitner <
> >> ghagleit...@hortonworks.com> wrote:
> >>
> >>> +1
> >>> 
> >>> From: Sankar Hariappan 
> >>> Sent: Tuesday, May 15, 2018 9:03 AM
> >>> To: dev@hive.apache.org
> >>> Subject: Re: [VOTE] Stricter commit guidelines
> >>>
> >>> +1
> >>>
> >>>
> >>> On 15/05/18, 9:30 PM, "Sahil Takiar"  wrote:
> >>>
>  +1
> 
>  On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley <
> owen.omal...@gmail.com
> >>>
>  wrote:
> 
> > +1
> >
> > On Tue, May 15, 2018 at 8:55 AM, Peter Vary 
> >> wrote:
> >
> >> +1 - Hoping for something like this for a long while! Thanks for
> >>> taking
> >> this up all!
> >>
> >>> On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
> >> jcama...@apache.org> wrote:
> >>>
> >>> Forgot to mention the length of the vote in original message.
> >>>
> >>> Let's leave the vote open for a shorter period than usual, for
> >>> instance
> >> 48 hours, i.e., till Wednesday 10pm PST. Situation can only get
> >> worse
> > than
> >> it is now if we do not take action for a longer period.
> >>>
> >>> As Alan suggested, vote passes if there is a lazy majority (at
> >>> least 3
> >> votes, more +1s than -1s).
> >>>
> >>> Thanks,
> >>> Jesús
> >>>
> >>>
> >>> On 5/15/18, 8:37 AM, "Andrew Sherman" 
> >>> wrote:
> >>>
> >>>   +1
> >>>
> >>>   On Tue, May 15, 2018 at 2:34 AM Rui Li 
> > wrote:
> >>>
>  +1
> 
>  On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
>  pjayachand...@hortonworks.com> wrote:
> 
> > +1
> >
> >
> >
> > Thanks
> > Prasanth
> >
> >
> >
> > On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho
> >> Rodriguez"
> >>> <
> > jcama...@apache.org> wrote:
> >
> >
> > After work has been done to ignore most of the tests that were
> > failing
> > consistently/intermittently [1], I wanted to start this vote to
> > gather
> > support from the community to be stricter wrt committing patches
> >>> to
> >> Hive.
> > The committers guide [2] already specifies that a +1 should be
> > obtained
> > before committing, but there is another clause that allows
> >>> committing
>  under
> > the presence of flaky tests (clause 4). Flaky tests are as good
> >> as
> >> having
> > no tests, hence I propose to remove clause 4 and enforce the +1
> >>> from
> > testing infra before committing.
> >
> >
> >
> > As I see it, by enforcing that we always get a +1 from the
> >> testing
> >> infra
> > before committing, 1) we will have a more stable project, and 2)
> >>> we
> >> will
> > have another incentive as a community to create a more robust
> >>> testing
> > infra, e.g., replacing flaky tests for similar unit tests that
> >> are
> > not
> > flaky, trying to decrease running time for tests, etc.
> >
> >
> >
> > Please, share your thoughts about this.
> >
> >
> >
> > Here is my +1.
> >
> >
> >
> > Thanks,
> >
> > Jes?s
> >
> >
> >
> > [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> > mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
> >
> > [2] https://cwiki.apache.org/confluence/display/Hive/
> > HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
> >
> >
> >
> >
> >
> 
> 
>  --
>  Best regards!
>  Rui Li
> 
> >>>
> >>>
> >>>
> >>
> >>
> >
> 
> 
> 
>  --
>  Sahil Takiar
>  Software Engineer
>  takiar.sa...@gmail.com | (510) 673-0309
> >>>
> >>>
> >>>
> >>
>
>

Re: [VOTE] Stricter commit guidelines

[Vineet Garg]

+1

> On May 15, 2018, at 9:13 AM, Alan Gates  wrote:
> 
> +1.
> 
> Alan.
> 
> On Tue, May 15, 2018 at 9:12 AM, Sergio Pena 
> wrote:
> 
>> +1
>> 
>> On Tue, May 15, 2018 at 11:05 AM, Gunther Hagleitner <
>> ghagleit...@hortonworks.com> wrote:
>> 
>>> +1
>>> 
>>> From: Sankar Hariappan 
>>> Sent: Tuesday, May 15, 2018 9:03 AM
>>> To: dev@hive.apache.org
>>> Subject: Re: [VOTE] Stricter commit guidelines
>>> 
>>> +1
>>> 
>>> 
>>> On 15/05/18, 9:30 PM, "Sahil Takiar"  wrote:
>>> 
 +1
 
 On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley >> 
 wrote:
 
> +1
> 
> On Tue, May 15, 2018 at 8:55 AM, Peter Vary 
>> wrote:
> 
>> +1 - Hoping for something like this for a long while! Thanks for
>>> taking
>> this up all!
>> 
>>> On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
>> jcama...@apache.org> wrote:
>>> 
>>> Forgot to mention the length of the vote in original message.
>>> 
>>> Let's leave the vote open for a shorter period than usual, for
>>> instance
>> 48 hours, i.e., till Wednesday 10pm PST. Situation can only get
>> worse
> than
>> it is now if we do not take action for a longer period.
>>> 
>>> As Alan suggested, vote passes if there is a lazy majority (at
>>> least 3
>> votes, more +1s than -1s).
>>> 
>>> Thanks,
>>> Jesús
>>> 
>>> 
>>> On 5/15/18, 8:37 AM, "Andrew Sherman" 
>>> wrote:
>>> 
>>>   +1
>>> 
>>>   On Tue, May 15, 2018 at 2:34 AM Rui Li 
> wrote:
>>> 
 +1
 
 On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
 pjayachand...@hortonworks.com> wrote:
 
> +1
> 
> 
> 
> Thanks
> Prasanth
> 
> 
> 
> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho
>> Rodriguez"
>>> <
> jcama...@apache.org> wrote:
> 
> 
> After work has been done to ignore most of the tests that were
> failing
> consistently/intermittently [1], I wanted to start this vote to
> gather
> support from the community to be stricter wrt committing patches
>>> to
>> Hive.
> The committers guide [2] already specifies that a +1 should be
> obtained
> before committing, but there is another clause that allows
>>> committing
 under
> the presence of flaky tests (clause 4). Flaky tests are as good
>> as
>> having
> no tests, hence I propose to remove clause 4 and enforce the +1
>>> from
> testing infra before committing.
> 
> 
> 
> As I see it, by enforcing that we always get a +1 from the
>> testing
>> infra
> before committing, 1) we will have a more stable project, and 2)
>>> we
>> will
> have another incentive as a community to create a more robust
>>> testing
> infra, e.g., replacing flaky tests for similar unit tests that
>> are
> not
> flaky, trying to decrease running time for tests, etc.
> 
> 
> 
> Please, share your thoughts about this.
> 
> 
> 
> Here is my +1.
> 
> 
> 
> Thanks,
> 
> Jes?s
> 
> 
> 
> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
> 
> [2] https://cwiki.apache.org/confluence/display/Hive/
> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
> 
> 
> 
> 
> 
 
 
 --
 Best regards!
 Rui Li
 
>>> 
>>> 
>>> 
>> 
>> 
> 
 
 
 
 --
 Sahil Takiar
 Software Engineer
 takiar.sa...@gmail.com | (510) 673-0309
>>> 
>>> 
>>> 
>> 

Review Request 67138: HIVE-4367 enhance TRUNCATE syntex to drop data of external table

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67138/
---

Review request for hive and Teddy Choi.


Bugs: HIVE-4367
https://issues.apache.org/jira/browse/HIVE-4367


Repository: hive-git


Description
---

Allow TRUNCATE TABLE for external tables with FORCE option


Diffs
-

  itests/src/test/resources/testconfiguration.properties cf6d19a593 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
f0b9edaf01 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 09a4368984 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 3712a53521 
  ql/src/test/queries/clientpositive/truncate_external_force.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/truncate_external_force.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/67138/diff/1/


Testing
---

qtest


Thanks,

Jason Dere

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

[Andrew Sherman via Review Board]

> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > 
> >
> > why would we want the ec commands to work outside the 
> > `TestErasureCodingHDFSCliDriver`?
> 
> Andrew Sherman wrote:
> So you could run an ec test in TestCliDriver (sorry reviewboard lost my 
> ealrier reply)
> 
> Sahil Takiar wrote:
> why would you want to do that? whats the use case? shouldn't 
> `TestErasureCodingHDFSCliDriver` encapsulate all EC-related tests? plus I'm 
> not sure how you could run any EC commands against a local filesystem, 
> wouldn't they all be no-ops?

They would. When I'm developing an EC test I may sometimes want to run the test 
in TestCLiDriver with erasure commands being no-ops as a way to validate the 
script. Myabe this is weird, but I've been doing it.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

[Andrew Sherman via Review Board]

> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
> looks like it does already
> 
> Sahil Takiar wrote:
> then is the tez specific code necessary?

I think it is, as tez has its own hive-site.xml but I am not 100% sure


> On May 10, 2018, noon, Sahil Takiar wrote:
> > testutils/ptest2/conf/deployed/master-mr2.properties
> > Line 75 (original), 75 (patched)
> > 
> >
> > have you manually deployed these changes to the ptest server? this file 
> > is just a copy of whats already been deployed, so its just for reference
> > 
> > also, why skip batching?
> 
> Andrew Sherman wrote:
> OK I have no idea what ut.itests.qtest.skipBatching means I just copied 
> TestEncryptedHDFSCliDriver :-(
> 
> As for deploying these changes, I don't know what that means, my new 
> tests did appear to run. can you explain more?
> 
> Sahil Takiar wrote:
> anytime you add a new cli driver, you have to manually modify a file on 
> the ptest master server, you have to modify the file 
> `/usr/local/hiveptest/profiles/master-mr2.properties` you probably don't have 
> permissions though, so let me know the final diff for the this file and I can 
> deploy it.
> 
> can we check if batching can be used for this? i think batching means 
> that q-tests get bundled together into a "batch" of q-tests that are run in a 
> single `mvn test` command

Batching works locally for me so I have removed this. I'll provide a patch for 
master-mr2.properties before commit


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   

[jira] [Created] (HIVE-19558) HiveAuthorizationProviderBase gets catalog name from config rather than db object

[Alan Gates (JIRA)]

Alan Gates created HIVE-19558:
-

 Summary: HiveAuthorizationProviderBase gets catalog name from 
config rather than db object
 Key: HIVE-19558
 URL: https://issues.apache.org/jira/browse/HIVE-19558
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 3.0.0
Reporter: Alan Gates
Assignee: Alan Gates
 Fix For: 3.0.1


HiveAuthorizationProviderBase.getDatabase uses just the database name to fetch 
the database, relying on getDefaultCatalog() to fetch the catalog name from the 
conf file.  This does not work when the client has passed in an object for a 
different catalog.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Stricter commit guidelines

[Alan Gates]

+1.

Alan.

On Tue, May 15, 2018 at 9:12 AM, Sergio Pena 
wrote:

> +1
>
> On Tue, May 15, 2018 at 11:05 AM, Gunther Hagleitner <
> ghagleit...@hortonworks.com> wrote:
>
> > +1
> > 
> > From: Sankar Hariappan 
> > Sent: Tuesday, May 15, 2018 9:03 AM
> > To: dev@hive.apache.org
> > Subject: Re: [VOTE] Stricter commit guidelines
> >
> > +1
> >
> >
> > On 15/05/18, 9:30 PM, "Sahil Takiar"  wrote:
> >
> > >+1
> > >
> > >On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley  >
> > >wrote:
> > >
> > >> +1
> > >>
> > >> On Tue, May 15, 2018 at 8:55 AM, Peter Vary 
> wrote:
> > >>
> > >> > +1 - Hoping for something like this for a long while! Thanks for
> > taking
> > >> > this up all!
> > >> >
> > >> > > On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
> > >> > jcama...@apache.org> wrote:
> > >> > >
> > >> > > Forgot to mention the length of the vote in original message.
> > >> > >
> > >> > > Let's leave the vote open for a shorter period than usual, for
> > instance
> > >> > 48 hours, i.e., till Wednesday 10pm PST. Situation can only get
> worse
> > >> than
> > >> > it is now if we do not take action for a longer period.
> > >> > >
> > >> > > As Alan suggested, vote passes if there is a lazy majority (at
> > least 3
> > >> > votes, more +1s than -1s).
> > >> > >
> > >> > > Thanks,
> > >> > > Jesús
> > >> > >
> > >> > >
> > >> > > On 5/15/18, 8:37 AM, "Andrew Sherman" 
> > wrote:
> > >> > >
> > >> > >+1
> > >> > >
> > >> > >On Tue, May 15, 2018 at 2:34 AM Rui Li 
> > >> wrote:
> > >> > >
> > >> > >> +1
> > >> > >>
> > >> > >> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
> > >> > >> pjayachand...@hortonworks.com> wrote:
> > >> > >>
> > >> > >>> +1
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>> Thanks
> > >> > >>> Prasanth
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho
> Rodriguez"
> > <
> > >> > >>> jcama...@apache.org> wrote:
> > >> > >>>
> > >> > >>>
> > >> > >>> After work has been done to ignore most of the tests that were
> > >> failing
> > >> > >>> consistently/intermittently [1], I wanted to start this vote to
> > >> gather
> > >> > >>> support from the community to be stricter wrt committing patches
> > to
> > >> > Hive.
> > >> > >>> The committers guide [2] already specifies that a +1 should be
> > >> obtained
> > >> > >>> before committing, but there is another clause that allows
> > committing
> > >> > >> under
> > >> > >>> the presence of flaky tests (clause 4). Flaky tests are as good
> as
> > >> > having
> > >> > >>> no tests, hence I propose to remove clause 4 and enforce the +1
> > from
> > >> > >>> testing infra before committing.
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>> As I see it, by enforcing that we always get a +1 from the
> testing
> > >> > infra
> > >> > >>> before committing, 1) we will have a more stable project, and 2)
> > we
> > >> > will
> > >> > >>> have another incentive as a community to create a more robust
> > testing
> > >> > >>> infra, e.g., replacing flaky tests for similar unit tests that
> are
> > >> not
> > >> > >>> flaky, trying to decrease running time for tests, etc.
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>> Please, share your thoughts about this.
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>> Here is my +1.
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>> Thanks,
> > >> > >>>
> > >> > >>> Jes?s
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> > >> > >>> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
> > >> > >>>
> > >> > >>> [2] https://cwiki.apache.org/confluence/display/Hive/
> > >> > >>> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>>
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> Best regards!
> > >> > >> Rui Li
> > >> > >>
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> > >--
> > >Sahil Takiar
> > >Software Engineer
> > >takiar.sa...@gmail.com | (510) 673-0309
> >
> >
> >
>

Re: [VOTE] Stricter commit guidelines

[Sergio Pena]

+1

On Tue, May 15, 2018 at 11:05 AM, Gunther Hagleitner <
ghagleit...@hortonworks.com> wrote:

> +1
> 
> From: Sankar Hariappan 
> Sent: Tuesday, May 15, 2018 9:03 AM
> To: dev@hive.apache.org
> Subject: Re: [VOTE] Stricter commit guidelines
>
> +1
>
>
> On 15/05/18, 9:30 PM, "Sahil Takiar"  wrote:
>
> >+1
> >
> >On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley 
> >wrote:
> >
> >> +1
> >>
> >> On Tue, May 15, 2018 at 8:55 AM, Peter Vary  wrote:
> >>
> >> > +1 - Hoping for something like this for a long while! Thanks for
> taking
> >> > this up all!
> >> >
> >> > > On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
> >> > jcama...@apache.org> wrote:
> >> > >
> >> > > Forgot to mention the length of the vote in original message.
> >> > >
> >> > > Let's leave the vote open for a shorter period than usual, for
> instance
> >> > 48 hours, i.e., till Wednesday 10pm PST. Situation can only get worse
> >> than
> >> > it is now if we do not take action for a longer period.
> >> > >
> >> > > As Alan suggested, vote passes if there is a lazy majority (at
> least 3
> >> > votes, more +1s than -1s).
> >> > >
> >> > > Thanks,
> >> > > Jesús
> >> > >
> >> > >
> >> > > On 5/15/18, 8:37 AM, "Andrew Sherman" 
> wrote:
> >> > >
> >> > >+1
> >> > >
> >> > >On Tue, May 15, 2018 at 2:34 AM Rui Li 
> >> wrote:
> >> > >
> >> > >> +1
> >> > >>
> >> > >> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
> >> > >> pjayachand...@hortonworks.com> wrote:
> >> > >>
> >> > >>> +1
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> Thanks
> >> > >>> Prasanth
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez"
> <
> >> > >>> jcama...@apache.org> wrote:
> >> > >>>
> >> > >>>
> >> > >>> After work has been done to ignore most of the tests that were
> >> failing
> >> > >>> consistently/intermittently [1], I wanted to start this vote to
> >> gather
> >> > >>> support from the community to be stricter wrt committing patches
> to
> >> > Hive.
> >> > >>> The committers guide [2] already specifies that a +1 should be
> >> obtained
> >> > >>> before committing, but there is another clause that allows
> committing
> >> > >> under
> >> > >>> the presence of flaky tests (clause 4). Flaky tests are as good as
> >> > having
> >> > >>> no tests, hence I propose to remove clause 4 and enforce the +1
> from
> >> > >>> testing infra before committing.
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> As I see it, by enforcing that we always get a +1 from the testing
> >> > infra
> >> > >>> before committing, 1) we will have a more stable project, and 2)
> we
> >> > will
> >> > >>> have another incentive as a community to create a more robust
> testing
> >> > >>> infra, e.g., replacing flaky tests for similar unit tests that are
> >> not
> >> > >>> flaky, trying to decrease running time for tests, etc.
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> Please, share your thoughts about this.
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> Here is my +1.
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> Thanks,
> >> > >>>
> >> > >>> Jes?s
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> >> > >>> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
> >> > >>>
> >> > >>> [2] https://cwiki.apache.org/confluence/display/Hive/
> >> > >>> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Best regards!
> >> > >> Rui Li
> >> > >>
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >
> >
> >
> >--
> >Sahil Takiar
> >Software Engineer
> >takiar.sa...@gmail.com | (510) 673-0309
>
>
>

Re: [VOTE] Stricter commit guidelines

[Gunther Hagleitner]

+1

From: Sankar Hariappan 
Sent: Tuesday, May 15, 2018 9:03 AM
To: dev@hive.apache.org
Subject: Re: [VOTE] Stricter commit guidelines

+1


On 15/05/18, 9:30 PM, "Sahil Takiar"  wrote:

>+1
>
>On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley 
>wrote:
>
>> +1
>>
>> On Tue, May 15, 2018 at 8:55 AM, Peter Vary  wrote:
>>
>> > +1 - Hoping for something like this for a long while! Thanks for taking
>> > this up all!
>> >
>> > > On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
>> > jcama...@apache.org> wrote:
>> > >
>> > > Forgot to mention the length of the vote in original message.
>> > >
>> > > Let's leave the vote open for a shorter period than usual, for instance
>> > 48 hours, i.e., till Wednesday 10pm PST. Situation can only get worse
>> than
>> > it is now if we do not take action for a longer period.
>> > >
>> > > As Alan suggested, vote passes if there is a lazy majority (at least 3
>> > votes, more +1s than -1s).
>> > >
>> > > Thanks,
>> > > Jesús
>> > >
>> > >
>> > > On 5/15/18, 8:37 AM, "Andrew Sherman"  wrote:
>> > >
>> > >+1
>> > >
>> > >On Tue, May 15, 2018 at 2:34 AM Rui Li 
>> wrote:
>> > >
>> > >> +1
>> > >>
>> > >> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
>> > >> pjayachand...@hortonworks.com> wrote:
>> > >>
>> > >>> +1
>> > >>>
>> > >>>
>> > >>>
>> > >>> Thanks
>> > >>> Prasanth
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
>> > >>> jcama...@apache.org> wrote:
>> > >>>
>> > >>>
>> > >>> After work has been done to ignore most of the tests that were
>> failing
>> > >>> consistently/intermittently [1], I wanted to start this vote to
>> gather
>> > >>> support from the community to be stricter wrt committing patches to
>> > Hive.
>> > >>> The committers guide [2] already specifies that a +1 should be
>> obtained
>> > >>> before committing, but there is another clause that allows committing
>> > >> under
>> > >>> the presence of flaky tests (clause 4). Flaky tests are as good as
>> > having
>> > >>> no tests, hence I propose to remove clause 4 and enforce the +1 from
>> > >>> testing infra before committing.
>> > >>>
>> > >>>
>> > >>>
>> > >>> As I see it, by enforcing that we always get a +1 from the testing
>> > infra
>> > >>> before committing, 1) we will have a more stable project, and 2) we
>> > will
>> > >>> have another incentive as a community to create a more robust testing
>> > >>> infra, e.g., replacing flaky tests for similar unit tests that are
>> not
>> > >>> flaky, trying to decrease running time for tests, etc.
>> > >>>
>> > >>>
>> > >>>
>> > >>> Please, share your thoughts about this.
>> > >>>
>> > >>>
>> > >>>
>> > >>> Here is my +1.
>> > >>>
>> > >>>
>> > >>>
>> > >>> Thanks,
>> > >>>
>> > >>> Jes?s
>> > >>>
>> > >>>
>> > >>>
>> > >>> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
>> > >>> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
>> > >>>
>> > >>> [2] https://cwiki.apache.org/confluence/display/Hive/
>> > >>> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>
>> > >>
>> > >> --
>> > >> Best regards!
>> > >> Rui Li
>> > >>
>> > >
>> > >
>> > >
>> >
>> >
>>
>
>
>
>--
>Sahil Takiar
>Software Engineer
>takiar.sa...@gmail.com | (510) 673-0309

Re: [VOTE] Stricter commit guidelines

[Sankar Hariappan]

+1


On 15/05/18, 9:30 PM, "Sahil Takiar"  wrote:

>+1
>
>On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley 
>wrote:
>
>> +1
>>
>> On Tue, May 15, 2018 at 8:55 AM, Peter Vary  wrote:
>>
>> > +1 - Hoping for something like this for a long while! Thanks for taking
>> > this up all!
>> >
>> > > On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
>> > jcama...@apache.org> wrote:
>> > >
>> > > Forgot to mention the length of the vote in original message.
>> > >
>> > > Let's leave the vote open for a shorter period than usual, for instance
>> > 48 hours, i.e., till Wednesday 10pm PST. Situation can only get worse
>> than
>> > it is now if we do not take action for a longer period.
>> > >
>> > > As Alan suggested, vote passes if there is a lazy majority (at least 3
>> > votes, more +1s than -1s).
>> > >
>> > > Thanks,
>> > > Jesús
>> > >
>> > >
>> > > On 5/15/18, 8:37 AM, "Andrew Sherman"  wrote:
>> > >
>> > >+1
>> > >
>> > >On Tue, May 15, 2018 at 2:34 AM Rui Li 
>> wrote:
>> > >
>> > >> +1
>> > >>
>> > >> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
>> > >> pjayachand...@hortonworks.com> wrote:
>> > >>
>> > >>> +1
>> > >>>
>> > >>>
>> > >>>
>> > >>> Thanks
>> > >>> Prasanth
>> > >>>
>> > >>>
>> > >>>
>> > >>> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
>> > >>> jcama...@apache.org> wrote:
>> > >>>
>> > >>>
>> > >>> After work has been done to ignore most of the tests that were
>> failing
>> > >>> consistently/intermittently [1], I wanted to start this vote to
>> gather
>> > >>> support from the community to be stricter wrt committing patches to
>> > Hive.
>> > >>> The committers guide [2] already specifies that a +1 should be
>> obtained
>> > >>> before committing, but there is another clause that allows committing
>> > >> under
>> > >>> the presence of flaky tests (clause 4). Flaky tests are as good as
>> > having
>> > >>> no tests, hence I propose to remove clause 4 and enforce the +1 from
>> > >>> testing infra before committing.
>> > >>>
>> > >>>
>> > >>>
>> > >>> As I see it, by enforcing that we always get a +1 from the testing
>> > infra
>> > >>> before committing, 1) we will have a more stable project, and 2) we
>> > will
>> > >>> have another incentive as a community to create a more robust testing
>> > >>> infra, e.g., replacing flaky tests for similar unit tests that are
>> not
>> > >>> flaky, trying to decrease running time for tests, etc.
>> > >>>
>> > >>>
>> > >>>
>> > >>> Please, share your thoughts about this.
>> > >>>
>> > >>>
>> > >>>
>> > >>> Here is my +1.
>> > >>>
>> > >>>
>> > >>>
>> > >>> Thanks,
>> > >>>
>> > >>> Jes?s
>> > >>>
>> > >>>
>> > >>>
>> > >>> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
>> > >>> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
>> > >>>
>> > >>> [2] https://cwiki.apache.org/confluence/display/Hive/
>> > >>> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>
>> > >>
>> > >> --
>> > >> Best regards!
>> > >> Rui Li
>> > >>
>> > >
>> > >
>> > >
>> >
>> >
>>
>
>
>
>-- 
>Sahil Takiar
>Software Engineer
>takiar.sa...@gmail.com | (510) 673-0309

Re: [VOTE] Stricter commit guidelines

[Sahil Takiar]

+1

On Tue, May 15, 2018 at 10:56 AM, Owen O'Malley 
wrote:

> +1
>
> On Tue, May 15, 2018 at 8:55 AM, Peter Vary  wrote:
>
> > +1 - Hoping for something like this for a long while! Thanks for taking
> > this up all!
> >
> > > On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
> > jcama...@apache.org> wrote:
> > >
> > > Forgot to mention the length of the vote in original message.
> > >
> > > Let's leave the vote open for a shorter period than usual, for instance
> > 48 hours, i.e., till Wednesday 10pm PST. Situation can only get worse
> than
> > it is now if we do not take action for a longer period.
> > >
> > > As Alan suggested, vote passes if there is a lazy majority (at least 3
> > votes, more +1s than -1s).
> > >
> > > Thanks,
> > > Jesús
> > >
> > >
> > > On 5/15/18, 8:37 AM, "Andrew Sherman"  wrote:
> > >
> > >+1
> > >
> > >On Tue, May 15, 2018 at 2:34 AM Rui Li 
> wrote:
> > >
> > >> +1
> > >>
> > >> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
> > >> pjayachand...@hortonworks.com> wrote:
> > >>
> > >>> +1
> > >>>
> > >>>
> > >>>
> > >>> Thanks
> > >>> Prasanth
> > >>>
> > >>>
> > >>>
> > >>> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
> > >>> jcama...@apache.org> wrote:
> > >>>
> > >>>
> > >>> After work has been done to ignore most of the tests that were
> failing
> > >>> consistently/intermittently [1], I wanted to start this vote to
> gather
> > >>> support from the community to be stricter wrt committing patches to
> > Hive.
> > >>> The committers guide [2] already specifies that a +1 should be
> obtained
> > >>> before committing, but there is another clause that allows committing
> > >> under
> > >>> the presence of flaky tests (clause 4). Flaky tests are as good as
> > having
> > >>> no tests, hence I propose to remove clause 4 and enforce the +1 from
> > >>> testing infra before committing.
> > >>>
> > >>>
> > >>>
> > >>> As I see it, by enforcing that we always get a +1 from the testing
> > infra
> > >>> before committing, 1) we will have a more stable project, and 2) we
> > will
> > >>> have another incentive as a community to create a more robust testing
> > >>> infra, e.g., replacing flaky tests for similar unit tests that are
> not
> > >>> flaky, trying to decrease running time for tests, etc.
> > >>>
> > >>>
> > >>>
> > >>> Please, share your thoughts about this.
> > >>>
> > >>>
> > >>>
> > >>> Here is my +1.
> > >>>
> > >>>
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jes?s
> > >>>
> > >>>
> > >>>
> > >>> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> > >>> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
> > >>>
> > >>> [2] https://cwiki.apache.org/confluence/display/Hive/
> > >>> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Best regards!
> > >> Rui Li
> > >>
> > >
> > >
> > >
> >
> >
>



-- 
Sahil Takiar
Software Engineer
takiar.sa...@gmail.com | (510) 673-0309

Re: [VOTE] Stricter commit guidelines

[Owen O'Malley]

+1

On Tue, May 15, 2018 at 8:55 AM, Peter Vary  wrote:

> +1 - Hoping for something like this for a long while! Thanks for taking
> this up all!
>
> > On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
> >
> > Forgot to mention the length of the vote in original message.
> >
> > Let's leave the vote open for a shorter period than usual, for instance
> 48 hours, i.e., till Wednesday 10pm PST. Situation can only get worse than
> it is now if we do not take action for a longer period.
> >
> > As Alan suggested, vote passes if there is a lazy majority (at least 3
> votes, more +1s than -1s).
> >
> > Thanks,
> > Jesús
> >
> >
> > On 5/15/18, 8:37 AM, "Andrew Sherman"  wrote:
> >
> >+1
> >
> >On Tue, May 15, 2018 at 2:34 AM Rui Li  wrote:
> >
> >> +1
> >>
> >> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
> >> pjayachand...@hortonworks.com> wrote:
> >>
> >>> +1
> >>>
> >>>
> >>>
> >>> Thanks
> >>> Prasanth
> >>>
> >>>
> >>>
> >>> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
> >>> jcama...@apache.org> wrote:
> >>>
> >>>
> >>> After work has been done to ignore most of the tests that were failing
> >>> consistently/intermittently [1], I wanted to start this vote to gather
> >>> support from the community to be stricter wrt committing patches to
> Hive.
> >>> The committers guide [2] already specifies that a +1 should be obtained
> >>> before committing, but there is another clause that allows committing
> >> under
> >>> the presence of flaky tests (clause 4). Flaky tests are as good as
> having
> >>> no tests, hence I propose to remove clause 4 and enforce the +1 from
> >>> testing infra before committing.
> >>>
> >>>
> >>>
> >>> As I see it, by enforcing that we always get a +1 from the testing
> infra
> >>> before committing, 1) we will have a more stable project, and 2) we
> will
> >>> have another incentive as a community to create a more robust testing
> >>> infra, e.g., replacing flaky tests for similar unit tests that are not
> >>> flaky, trying to decrease running time for tests, etc.
> >>>
> >>>
> >>>
> >>> Please, share your thoughts about this.
> >>>
> >>>
> >>>
> >>> Here is my +1.
> >>>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Jes?s
> >>>
> >>>
> >>>
> >>> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> >>> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
> >>>
> >>> [2] https://cwiki.apache.org/confluence/display/Hive/
> >>> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Best regards!
> >> Rui Li
> >>
> >
> >
> >
>
>

Re: [VOTE] Stricter commit guidelines

[Peter Vary]

+1 - Hoping for something like this for a long while! Thanks for taking this up 
all!

> On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez  
> wrote:
> 
> Forgot to mention the length of the vote in original message.
> 
> Let's leave the vote open for a shorter period than usual, for instance 48 
> hours, i.e., till Wednesday 10pm PST. Situation can only get worse than it is 
> now if we do not take action for a longer period.
> 
> As Alan suggested, vote passes if there is a lazy majority (at least 3 votes, 
> more +1s than -1s).
> 
> Thanks,
> Jesús
> 
> 
> On 5/15/18, 8:37 AM, "Andrew Sherman"  wrote:
> 
>+1
> 
>On Tue, May 15, 2018 at 2:34 AM Rui Li  wrote:
> 
>> +1
>> 
>> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>> +1
>>> 
>>> 
>>> 
>>> Thanks
>>> Prasanth
>>> 
>>> 
>>> 
>>> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
>>> jcama...@apache.org> wrote:
>>> 
>>> 
>>> After work has been done to ignore most of the tests that were failing
>>> consistently/intermittently [1], I wanted to start this vote to gather
>>> support from the community to be stricter wrt committing patches to Hive.
>>> The committers guide [2] already specifies that a +1 should be obtained
>>> before committing, but there is another clause that allows committing
>> under
>>> the presence of flaky tests (clause 4). Flaky tests are as good as having
>>> no tests, hence I propose to remove clause 4 and enforce the +1 from
>>> testing infra before committing.
>>> 
>>> 
>>> 
>>> As I see it, by enforcing that we always get a +1 from the testing infra
>>> before committing, 1) we will have a more stable project, and 2) we will
>>> have another incentive as a community to create a more robust testing
>>> infra, e.g., replacing flaky tests for similar unit tests that are not
>>> flaky, trying to decrease running time for tests, etc.
>>> 
>>> 
>>> 
>>> Please, share your thoughts about this.
>>> 
>>> 
>>> 
>>> Here is my +1.
>>> 
>>> 
>>> 
>>> Thanks,
>>> 
>>> Jes?s
>>> 
>>> 
>>> 
>>> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
>>> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
>>> 
>>> [2] https://cwiki.apache.org/confluence/display/Hive/
>>> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
>> --
>> Best regards!
>> Rui Li
>> 
> 
> 
> 

Re: [VOTE] Stricter commit guidelines

[Jesus Camacho Rodriguez]

Forgot to mention the length of the vote in original message.

Let's leave the vote open for a shorter period than usual, for instance 48 
hours, i.e., till Wednesday 10pm PST. Situation can only get worse than it is 
now if we do not take action for a longer period.

As Alan suggested, vote passes if there is a lazy majority (at least 3 votes, 
more +1s than -1s).

Thanks,
Jesús


On 5/15/18, 8:37 AM, "Andrew Sherman"  wrote:

+1

On Tue, May 15, 2018 at 2:34 AM Rui Li  wrote:

> +1
>
> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
>
> > +1
> >
> >
> >
> > Thanks
> > Prasanth
> >
> >
> >
> > On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
> > jcama...@apache.org> wrote:
> >
> >
> > After work has been done to ignore most of the tests that were failing
> > consistently/intermittently [1], I wanted to start this vote to gather
> > support from the community to be stricter wrt committing patches to 
Hive.
> > The committers guide [2] already specifies that a +1 should be obtained
> > before committing, but there is another clause that allows committing
> under
> > the presence of flaky tests (clause 4). Flaky tests are as good as 
having
> > no tests, hence I propose to remove clause 4 and enforce the +1 from
> > testing infra before committing.
> >
> >
> >
> > As I see it, by enforcing that we always get a +1 from the testing infra
> > before committing, 1) we will have a more stable project, and 2) we will
> > have another incentive as a community to create a more robust testing
> > infra, e.g., replacing flaky tests for similar unit tests that are not
> > flaky, trying to decrease running time for tests, etc.
> >
> >
> >
> > Please, share your thoughts about this.
> >
> >
> >
> > Here is my +1.
> >
> >
> >
> > Thanks,
> >
> > Jes?s
> >
> >
> >
> > [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> > mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
> >
> > [2] https://cwiki.apache.org/confluence/display/Hive/
> > HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
> >
> >
> >
> >
> >
>
>
> --
> Best regards!
> Rui Li
>

Re: [VOTE] Stricter commit guidelines

[Andrew Sherman]

+1

On Tue, May 15, 2018 at 2:34 AM Rui Li  wrote:

> +1
>
> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
>
> > +1
> >
> >
> >
> > Thanks
> > Prasanth
> >
> >
> >
> > On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
> > jcama...@apache.org> wrote:
> >
> >
> > After work has been done to ignore most of the tests that were failing
> > consistently/intermittently [1], I wanted to start this vote to gather
> > support from the community to be stricter wrt committing patches to Hive.
> > The committers guide [2] already specifies that a +1 should be obtained
> > before committing, but there is another clause that allows committing
> under
> > the presence of flaky tests (clause 4). Flaky tests are as good as having
> > no tests, hence I propose to remove clause 4 and enforce the +1 from
> > testing infra before committing.
> >
> >
> >
> > As I see it, by enforcing that we always get a +1 from the testing infra
> > before committing, 1) we will have a more stable project, and 2) we will
> > have another incentive as a community to create a more robust testing
> > infra, e.g., replacing flaky tests for similar unit tests that are not
> > flaky, trying to decrease running time for tests, etc.
> >
> >
> >
> > Please, share your thoughts about this.
> >
> >
> >
> > Here is my +1.
> >
> >
> >
> > Thanks,
> >
> > Jes?s
> >
> >
> >
> > [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> > mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
> >
> > [2] https://cwiki.apache.org/confluence/display/Hive/
> > HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
> >
> >
> >
> >
> >
>
>
> --
> Best regards!
> Rui Li
>

[jira] [Created] (HIVE-19557) stats: filters for dates are taking advantage of min/max values

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-19557:
---

 Summary: stats: filters for dates are taking advantage of min/max 
values
 Key: HIVE-19557
 URL: https://issues.apache.org/jira/browse/HIVE-19557
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


in StatsRulesProcFactory 
[https://github.com/apache/hive/blob/ab189f54047bbf6beeeaf8d0dcfd5fbe92e465fb/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L754|dates
 are assumed to be an integer]; however this is currently not true - and the 
resulting exception is handled as a default case... for N/3


{code}
set hive.explain.user=true;

create table d1(d date);
--  tblproperties('transactional'='false');

insert into d1 values
('2010-10-01'),
('2010-10-02'),
('2010-10-03'),
('2010-10-04'),
('2010-10-05'),
('2010-10-06'),
('2010-10-07'),
('2010-10-08'),
('2010-10-09'),
('2010-10-10');

analyze table d1 compute statistics for columns;

desc formatted d1;
desc formatted d1 d;

explain
select 'stats: FIL ~0 read',count(1) from d1 where d < '2010-03-01';

explain
select 'stats: FIL estimate some read',count(1) from d1 where d < '2010-10-03';

explain
select 'stats: FIL estimate all read',count(1) from d1 where d < '2010-11-03';
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

[Sahil Takiar]

> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
> looks like it does already

then is the tez specific code necessary?


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

[Sahil Takiar]

> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > 
> >
> > why would we want the ec commands to work outside the 
> > `TestErasureCodingHDFSCliDriver`?
> 
> Andrew Sherman wrote:
> So you could run an ec test in TestCliDriver (sorry reviewboard lost my 
> ealrier reply)

why would you want to do that? whats the use case? shouldn't 
`TestErasureCodingHDFSCliDriver` encapsulate all EC-related tests? plus I'm not 
sure how you could run any EC commands against a local filesystem, wouldn't 
they all be no-ops?


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 66290: HIVE-14388 : Add number of rows inserted message after insert command in Beeline

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66290/#review203121
---


Ship it!




+1 pending tests

- Peter Vary


On May 14, 2018, 8:07 p.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66290/
> ---
> 
> (Updated May 14, 2018, 8:07 p.m.)
> 
> 
> Review request for hive, Sahil Takiar and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-14388
> https://issues.apache.org/jira/browse/HIVE-14388
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently, when you run insert command on beeline, it returns a message 
> saying "No rows affected .."
> A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)"
> 
> Added the numRows parameter as part of QueryState.
> Adding the numRows to the response as well to display in beeline.
> 
> Getting the count in FileSinkOperator and setting it in statsMap, when it 
> operates only on table specific rows for the particular operation. (so that 
> we can get only the insert to table count and avoid counting non-table 
> specific file-sink operations happening during query execution).
> 
> 
> Diffs
> -
> 
>   beeline/src/main/resources/BeeLine.properties 
> c41b3ed637e04d8d2d9800ad5e9284264f7e4055 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
> b217259553be472863cd33bb2259aa700e6c3528 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 
> 06542cee02e5dc4696f2621bb45cc4f24c67dfda 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
> 52799b30c39af2f192c4ae22ce7d68b403014183 
>   ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java 
> cf9c2273159c0d779ea90ad029613678fb0967a6 
>   ql/src/java/org/apache/hadoop/hive/ql/QueryState.java 
> 706c9ffa48b9c3b4a6fdaae78bab1d39c3d0efda 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
> 01a5b4c9c328cb034a613a1539cea2584e122fb4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 
> fcdc9967f12a454a9d3f31031e2261f264479118 
>   ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 
> 18f4c69a191bde3cae2d5efac5ef20fd0b1a9f0c 
>   ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 
> 28f376f8c4c2151383286e754447d1349050ef4e 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 
> 96819f4e1c446f6de423f99c7697d548ff5dbe06 
>   ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 
> d2fcdaa1bfba03e1f0e4191c8d056b05f334443d 
>   service-rpc/if/TCLIService.thrift 30f8af7f3e6e0598b410498782900ac27971aef0 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 
> 4321ad6d3c966d30f7a69552f91804cf2f1ba6c4 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 
> b2b62c71492b844f4439367364c5c81aa62f3908 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetOperationStatusResp.java
>  15e8220eb3eb12b72c7b64029410dced33bc0d72 
>   service-rpc/src/gen/thrift/gen-php/Types.php 
> abb7c1ff3a2c8b72dc97689758266b675880e32b 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py 
> 0f8fd0745be0f4ed9e96b7bbe0f092d03649bcdf 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 
> 60183dae9e9927bd09a9676e49eeb4aea2401737 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 
> c9914ba9bf8653cbcbca7d6612e98a64058c0fcc 
>   service/src/java/org/apache/hive/service/cli/OperationStatus.java 
> 52cc3ae4f26b990b3e4edb52d9de85b3cc25f269 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> 3706c72abc77ac8bd77947cc1c5d084ddf965e9f 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> c64c99120ad21ee98af81ec6659a2722e3e1d1c7 
> 
> 
> Diff: https://reviews.apache.org/r/66290/diff/10/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

Re: Review Request 67126: HIVE-19326: stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67126/#review203111
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
Line 1910 (original), 1910 (patched)


this change fixes the tez.merge.files case; because the only problem in 
that cases is that the second filesink is not gathering stats

my own opinion: is that gathering statistics has no real overhead (it 
writes a file)...I think by enabling it here and there it somewhat just adds 
complexity



ql/src/test/results/clientpositive/llap/union_stats.q.out
Lines 427 (patched)


FS_7 is present 2 times in this plan

operator ids are reused multiple times in queries like:

from (select * from src union all select * from src)s
insert overwrite table t1 select *
insert overwrite table t2 select *;

if I understand correctly actually the file sink id's are reused for in 
every union branch to do output.

HIVE-19237 should fix this; and probably also remove indexInTezUnion 
setters/etc


- Zoltan Haindrich


On May 15, 2018, 1:07 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67126/
> ---
> 
> (Updated May 15, 2018, 1:07 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-19326
> https://issues.apache.org/jira/browse/HIVE-19326
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> in queries like: INSERT ... SELECT ... UNION ALL SELECT ... 
> the stats are only collected for the first select
> 
> there are 2 issues fixed - which both resulted in the same result:
> 
> * statscollectors have overwritten eachothers result; because the filename 
> was only dependent from the resulting table name
> * in case tez.merge.files the 2. task have not been set to collect statistics
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 
> 13c08de3c57fdd7fcd4181814bb8e547c699b9f1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
> 01a5b4c9c328cb034a613a1539cea2584e122fb4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 
> 108bb57c4189b720e672eb6f09b1ef05f78448c2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 
> 07991811f92bc7accae2fde23244f424bdd64c6b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 605bb09caba3b60ad8d51e50dace70757ab80188 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsCollectionContext.java 
> 5c3328c63e8ca37e284e1dc1cdbee5969e185a80 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsPublisher.java 
> 902b37f7874dd5b1afaf8c8bb1259c6f0ddf817f 
>   ql/src/test/queries/clientpositive/union_fast_stats.q 
> d69bef3ac083d5a06acda9f47e5d2c1cbe2dfb69 
>   ql/src/test/queries/clientpositive/union_rowcounts.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/union_stats.q 
> 0e91c23fea475ec95a7fa67433707cb290b277a2 
>   ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 
> f8adcd4ba24f2122f7e7e20770e24a71cfb01a7e 
>   ql/src/test/results/clientpositive/llap/union_fast_stats.q.out 
> 4ca5f47a850ba62290c0845eb11a8d0a32780526 
>   ql/src/test/results/clientpositive/llap/union_rowcounts.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/union_stats.q.out 
> 5a088f40f5fea41d04a5b6cb6c12bf852a22f097 
>   ql/src/test/results/clientpositive/union_stats.q.out 
> 8bd3f44b6e276d6636082673084d66bba3b5c0d3 
> 
> 
> Diff: https://reviews.apache.org/r/67126/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Zoltan Haindrich
> 
>

Review Request 67126: HIVE-19326: stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67126/
---

Review request for hive, Ashutosh Chauhan and Sergey Shelukhin.


Bugs: HIVE-19326
https://issues.apache.org/jira/browse/HIVE-19326


Repository: hive-git


Description
---

in queries like: INSERT ... SELECT ... UNION ALL SELECT ... 
the stats are only collected for the first select

there are 2 issues fixed - which both resulted in the same result:

* statscollectors have overwritten eachothers result; because the filename was 
only dependent from the resulting table name
* in case tez.merge.files the 2. task have not been set to collect statistics


Diffs
-

  itests/src/test/resources/testconfiguration.properties 
13c08de3c57fdd7fcd4181814bb8e547c699b9f1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
01a5b4c9c328cb034a613a1539cea2584e122fb4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 
108bb57c4189b720e672eb6f09b1ef05f78448c2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 
07991811f92bc7accae2fde23244f424bdd64c6b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
605bb09caba3b60ad8d51e50dace70757ab80188 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsCollectionContext.java 
5c3328c63e8ca37e284e1dc1cdbee5969e185a80 
  ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsPublisher.java 
902b37f7874dd5b1afaf8c8bb1259c6f0ddf817f 
  ql/src/test/queries/clientpositive/union_fast_stats.q 
d69bef3ac083d5a06acda9f47e5d2c1cbe2dfb69 
  ql/src/test/queries/clientpositive/union_rowcounts.q PRE-CREATION 
  ql/src/test/queries/clientpositive/union_stats.q 
0e91c23fea475ec95a7fa67433707cb290b277a2 
  ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 
f8adcd4ba24f2122f7e7e20770e24a71cfb01a7e 
  ql/src/test/results/clientpositive/llap/union_fast_stats.q.out 
4ca5f47a850ba62290c0845eb11a8d0a32780526 
  ql/src/test/results/clientpositive/llap/union_rowcounts.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/union_stats.q.out 
5a088f40f5fea41d04a5b6cb6c12bf852a22f097 
  ql/src/test/results/clientpositive/union_stats.q.out 
8bd3f44b6e276d6636082673084d66bba3b5c0d3 


Diff: https://reviews.apache.org/r/67126/diff/1/


Testing
---


Thanks,

Zoltan Haindrich

Re: [VOTE] Stricter commit guidelines

[Rui Li]

+1

On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> +1
>
>
>
> Thanks
> Prasanth
>
>
>
> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
> jcama...@apache.org> wrote:
>
>
> After work has been done to ignore most of the tests that were failing
> consistently/intermittently [1], I wanted to start this vote to gather
> support from the community to be stricter wrt committing patches to Hive.
> The committers guide [2] already specifies that a +1 should be obtained
> before committing, but there is another clause that allows committing under
> the presence of flaky tests (clause 4). Flaky tests are as good as having
> no tests, hence I propose to remove clause 4 and enforce the +1 from
> testing infra before committing.
>
>
>
> As I see it, by enforcing that we always get a +1 from the testing infra
> before committing, 1) we will have a more stable project, and 2) we will
> have another incentive as a community to create a more robust testing
> infra, e.g., replacing flaky tests for similar unit tests that are not
> flaky, trying to decrease running time for tests, etc.
>
>
>
> Please, share your thoughts about this.
>
>
>
> Here is my +1.
>
>
>
> Thanks,
>
> Jes?s
>
>
>
> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
>
> [2] https://cwiki.apache.org/confluence/display/Hive/
> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
>
>
>
>
>


-- 
Best regards!
Rui Li

[jira] [Created] (HIVE-19556) Ability to alter mapreduce job's name using hive config

[Kartik Bhatia (JIRA)]

Kartik Bhatia created HIVE-19556:


 Summary: Ability to alter mapreduce job's name using hive config
 Key: HIVE-19556
 URL: https://issues.apache.org/jira/browse/HIVE-19556
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Kartik Bhatia
Assignee: Kartik Bhatia


Currently hive by default sets job name as queryId + (Stage) if no name is 
supplied by user.

We need a capability for enriching the mapred job name at run time by picking 
up a class that enriches the existing name as per its implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Unsustainable situation with ptests

[Prasanth Jayachandran]

Wondering if we can add a state transition from “Patch Available” to “Ready To 
Commit” which can only be triggered by ptest bot on green test run.

Thanks
Prasanth



On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" 
> wrote:


I have been working on fixing this situation while commits were still coming in.

All the tests that have been disabled are in:
https://issues.apache.org/jira/browse/HIVE-19509
I have created new issues to reenable each of them, they are linked to that 
issue.
Maybe I was slightly aggressive disabling some of the tests, however that 
seemed to be the only way to bring the tests failures with age count > 1 to 
zero.

Instead of starting a vote to freeze the commits in another thread, I will 
start a vote to be stricter wrt committing to master, i.e., only commit if we 
get a clean QA run.

We can discuss more about this issue over there.

Thanks,
Jesús



On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:

Can we please make this freeze conditional, i.e. we unfreeze automatically
after ptest is clean (as evidenced by the clean HiveQA run on a given
JIRA).

On 18/5/14, 15:16, "Alan Gates"  wrote:

>We should do it in a separate thread so that people can see it with the
>[VOTE] subject.  Some people use that as a filter in their email to know
>when to pay attention to things.
>
>Alan.
>
>On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
>pjayachand...@hortonworks.com> wrote:
>
>> Will there be a separate voting thread? Or the voting on this thread is
>> sufficient for lock down?
>>
>> Thanks
>> Prasanth
>>
>> > On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
>> >
>> > ​I see there's support for this, but people are still pouring in
>>commits.
>> > I proposed we have a quick vote on this to lock down the commits
>>until we
>> > get to green.  That way everyone knows we have drawn the line at a
>> specific
>> > point.  Any commits after that point would be reverted.  There isn't a
>> > category in the bylaws that fits this kind of vote but I suggest lazy
>> > majority as the most appropriate one (at least 3 votes, more +1s than
>> > -1s).
>> >
>> > Alan.​
>> >
>> > On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
>> vih...@cloudera.com>
>> > wrote:
>> >
>> >> I worked on a few quick-fix optimizations in Ptest infrastructure
>>over
>> the
>> >> weekend which reduced the execution run from ~90 min to ~70 min per
>> run. I
>> >> had to restart Ptest multiple times. I was resubmitting the patches
>> which
>> >> were in the queue manually, but I may have missed a few. In case you
>> have a
>> >> patch which is pending pre-commit and you don't see it in the queue,
>> please
>> >> submit it manually or let me know if you don't have access to the
>> jenkins
>> >> job. I will continue to work on the sub-tasks in HIVE-19425 and will
>>do
>> >> some maintenance next weekend as well.
>> >>
>> >> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
>> >> jcama...@apache.org> wrote:
>> >>
>> >>> Vineet has already been working on disabling those tests that were
>> timing
>> >>> out. I am working on disabling those that are generating different q
>> >> files
>> >>> consistently for last ptests n runs. I am keeping track of all these
>> >> tests
>> >>> in https://issues.apache.org/jira/browse/HIVE-19509.
>> >>>
>> >>> -Jesús
>> >>>
>> >>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
>> >>> pjayachand...@hortonworks.com> wrote:
>> >>>
>> >>>+1 on freezing commits until we get repetitive green tests. We
>> should
>> >>> probably disable (and remember in a jira to reenable then at later
>> point)
>> >>> tests that are flaky to get repetitive green test runs.
>> >>>
>> >>>Thanks
>> >>>Prasanth
>> >>>
>> >>>
>> >>>
>> >>>On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
>> >> lirui.fu...@gmail.com
>> >>> > wrote:
>> >>>
>> >>>
>> >>>+1 to freezing commits until we stabilize
>> >>>
>> >>>On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
>> >>>wrote:
>> >>>
>>  In order to understand the end-to-end precommit flow I would like
>> >> to
>> >>> get
>>  access to the PreCommit-HIVE-Build jenkins script. Does anyone one
>> >>> know how
>>  can I get that?
>> 
>>  On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
>>  jcama...@apache.org> wrote:
>> 
>> > Bq. For the short term green runs, I think we should @Ignore the
>> >>> tests
>> > which
>> > are known to be failing since many runs. They are anyways not
>> >> being
>> > 

Re: [VOTE] Stricter commit guidelines

[Prasanth Jayachandran]

+1



Thanks
Prasanth



On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" 
> wrote:


After work has been done to ignore most of the tests that were failing 
consistently/intermittently [1], I wanted to start this vote to gather support 
from the community to be stricter wrt committing patches to Hive. The 
committers guide [2] already specifies that a +1 should be obtained before 
committing, but there is another clause that allows committing under the 
presence of flaky tests (clause 4). Flaky tests are as good as having no tests, 
hence I propose to remove clause 4 and enforce the +1 from testing infra before 
committing.



As I see it, by enforcing that we always get a +1 from the testing infra before 
committing, 1) we will have a more stable project, and 2) we will have another 
incentive as a community to create a more robust testing infra, e.g., replacing 
flaky tests for similar unit tests that are not flaky, trying to decrease 
running time for tests, etc.



Please, share your thoughts about this.



Here is my +1.



Thanks,

Jes?s



[1] 
http://mail-archives.apache.org/mod_mbox/hive-dev/201805.mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E

[2] 
https://cwiki.apache.org/confluence/display/Hive/HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches

Re: [VOTE] Stricter commit guidelines

[Thejas Nair]

+1

On Mon, May 14, 2018 at 10:44 PM, Jesus Camacho Rodriguez
 wrote:
> After work has been done to ignore most of the tests that were failing 
> consistently/intermittently [1], I wanted to start this vote to gather 
> support from the community to be stricter wrt committing patches to Hive. The 
> committers guide [2] already specifies that a +1 should be obtained before 
> committing, but there is another clause that allows committing under the 
> presence of flaky tests (clause 4). Flaky tests are as good as having no 
> tests, hence I propose to remove clause 4 and enforce the +1 from testing 
> infra before committing.
>
>
>
> As I see it, by enforcing that we always get a +1 from the testing infra 
> before committing, 1) we will have a more stable project, and 2) we will have 
> another incentive as a community to create a more robust testing infra, e.g., 
> replacing flaky tests for similar unit tests that are not flaky, trying to 
> decrease running time for tests, etc.
>
>
>
> Please, share your thoughts about this.
>
>
>
> Here is my +1.
>
>
>
> Thanks,
>
> Jesús
>
>
>
> [1] 
> http://mail-archives.apache.org/mod_mbox/hive-dev/201805.mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
>
> [2] 
> https://cwiki.apache.org/confluence/display/Hive/HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
>
>
>