Re: Hive on Tez Counters

I'll let Hive folks answer the questions about the Hive counters. In terms of the CPU counter - that was a bug in Tez-0.4.0, which has been fixed in 0.5.0. COMMITTED_HEAP_BYTES just represents the memory available to the JVM (Runtime.getRuntime().totalMemory()). This will only vary if the VM is

Re: Hive on Tez Counters

regardless of whether you run with Tez or MR. We've removed some counters in hive 0.13 (HIVE-4518) - the specific one you're looking for might be in that list. Thanks, Gunther. On Thu, Aug 21, 2014 at 11:13 AM, Siddharth Seth ss...@apache.org wrote: I'll let Hive folks answer

Re: Review Request 37333: HIVE-11515 Still some possible race condition in DynamicPartitionPruner

/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicPartitionPruner.java (line 472) https://reviews.apache.org/r/37333/#comment151511 Should be an error (IllegalStateException), instead of just logging this. - Siddharth Seth On Aug. 11, 2015, 12:49 a.m., Navis Ryu wrote

Re: [ANNOUNCE] New Hive Committer - Siddharth Seth

18 PM, Santlal J Gupta < > > santlal.gu...@bitwiseglobal.com> wrote: > > > >> Congratulations !! > >> > >> -Original Message- > >> From: Chetna C [mailto:chetna@gmail.com] > >> Sent: Thursday, October 22, 2015 8:58 AM > >> T

Re: Review Request 38783: HIVE-11969 start Tez session in background when starting CLI

aviour will be similar to what it is rightnow for invocation of open(). i.e. a RuntimeException will end up being thrown in case of most errors. Wondering if a SessionNotRunning exception from waitTillReady() needs to be handled differently in the async model. Separate jira anyway. - Siddhar

Re: unit tests in patches

Can a 'Target Version' field be added to jiras ? That would help to get rid of the confusion caused by Fix Version being used to represent branches a jira does go into. On Mon, Sep 21, 2015 at 12:55 PM, Ashutosh Chauhan wrote: > Hi everyone, > > Generally, its a good idea

Re: Review Request 40315: HIVE-12341 LLAP security

hive/llap/security/LlapSecurityHelper.java (line 66) <https://reviews.apache.org/r/40315/#comment168911> New jira to make this configurable - Siddharth Seth On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote: > > --- > This is an automat

Re: Review Request 40315: HIVE-12341 LLAP security

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/40315/#review109413 --- Ship it! Ship It! - Siddharth Seth On Dec. 5, 2015, 9:40 p.m

Re: Review Request 40315: HIVE-12341 LLAP security

creating new jiras. The compiled proto file seems to be missing from the RB patch. Assuming that'll be part of the commit. - Siddharth Seth On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically ge

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

h - without setting up explicit properties for tests only. - Siddharth Seth On Dec. 18, 2015, 10:10 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To

Re: Review Request 40315: HIVE-12341 LLAP security

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/40315/#review107877 --- - Siddharth Seth On Nov. 24, 2015, 11:11 p.m., Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

review comments in a bit. llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java (line 100) <https://reviews.apache.org/r/40315/#comment167199> Precondition check for security==true and token!=null - Siddharth Seth On Nov. 24, 2015, 11:11 p.m.,

Re: Review Request 40315: HIVE-12341 LLAP security

> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote: > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2361 > > <https://reviews.apache.org/r/40315/diff/2/?file=1127510#file1127510line2361> > > > > Don't think the default value

Re: Review Request 40315: HIVE-12341 LLAP security

ails of the ZKSecretManager - but it looks like the Tokens issued by any of the LLAP instances can be used by an application to communicate with all other instances. Also, are the tokens the same for different applications ? - Siddharth Seth On Nov. 16, 2015, 7:45 p.m.,

Re: Review Request 48186: HIVE-13443 LLAP: signing for the second state of submit (the event)

ive configuration) ql/src/java/org/apache/tez/dag/api/TaskSpecBuilder.java (line 66) <https://reviews.apache.org/r/48186/#comment201101> Please elaborate ? (This is not a public Tez API). Should this function just be removed. Used only once - the usage can be replaced by verte

Re: Review Request 48186: HIVE-13443 LLAP: signing for the second state of submit (the event)

- > > (Updated June 2, 2016, 8:55 p.m.) > > > Review request for hive and Siddharth Seth. > > > Repository: hive-git > > > Description > --- > > see jira > > > Diffs > - > > llap-client/src/java/org/apache/hadoop/hive/

Re: Review Request 48502: HIVE-13731 LLAP: return LLAP token with the splits

to stop sending this repeated information. Think there's a jira open to avoid sending the VertexSpec per split - that can include sending the token only once. - Siddharth Seth On June 9, 2016, 7:07 p.m., Sergey Shelukhin wrote

Re: [discuss] jdk8 support

Number of such tests > > > > >> should be relatively small. > > > > >> > > > > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal < > > mo...@cloudera.com > > > > > > > > >> wrote: > > > >

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

this will be caught by some tests. Would be good to add a couple of tests here as well. - Siddharth Seth On May 27, 2016, 9:32 p.m., Sergey Shelukhin wrote: > >

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46754/#review135722 --- Ship it! Ship It! - Siddharth Seth On May 28, 2016, 2:04

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

on the test before committing. - Siddharth Seth On May 27, 2016, 12:39 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

> Review request for hive, Jason Dere and Siddharth Seth. > > > Repository: hive-git > > > Description > --- > > see JIRA. Please ignore the first iteration, RB doesn't allow base patches on > submit > > > Diffs > - > > common/s

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

it's being checked in createSignedVertexSpec. The comment still seems incorrect though, since TaskSpec is never sent any longer. Assuming the serialized spec without a signature works in LLAP (i.e. when the daemon gets the VertexSpecBinary without a signature - it handles it correctly) - Siddha

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

the unnecessary TaskSpec creation, and potentially skipping the unnecessary VertexSpec being sent each time as a response from GenericUDTFGetSplits. - Siddharth Seth On June 2, 2016, midnight, Sergey Shelukhin wrote

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

> On May 31, 2016, 11:09 p.m., Siddharth Seth wrote: > > llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java, > > line 350 > > <https://reviews.apache.org/r/47563/diff/3/?file=1399778#file1399778line350> > > > > Is a signatu

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

> On May 31, 2016, 11:09 p.m., Siddharth Seth wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java, line > > 381 > > <https://reviews.apache.org/r/47563/diff/3/?file=1399784#file1399784line381> > > > > Not sure why the use

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

/ Testing --- Thanks, Siddharth Seth

Review Request 48818: See jira

/TestLlapTaskCommunicator.java 1901328 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java bdf254b Diff: https://reviews.apache.org/r/48818/diff/ Testing --- Thanks, Siddharth Seth

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

eful for other communication purposes There's nothing for it to do. It can be re-introduced if required ? (The Hook interface still exists in TezProcessor). - Siddharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48818/#review138141

Re: Hive 2.0 branch - starting next week, please ask RM before committing (in preparation for the release)

I'd like to get the changes under HIVE-12446 committed. The Tez 0.8.2 release is in progress and looks like it'll be available by early next week. 0.8.2 has some bug fixes, and support for features which allow better locality for LLAP in Hive 2.0. Thanks - Sid On Thu, Jan 14, 2016 at 11:15 AM,

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

/hadoop/hive/llap/tezplugins/TestLlapTaskCommunicator.java 1901328 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java bdf254b Diff: https://reviews.apache.org/r/48818/diff/ Testing --- Thanks, Siddharth Seth

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java bdf254b Diff: https://reviews.apache.org/r/48818/diff/ Testing --- Thanks, Siddharth Seth

Re: Hive 2.0 + tez setting up

As part of using Tez-0.8, you will need to update the tez tar which resides on HDFS. The path can be found out by looking at tez.lib.uris in tez-site.xml. You could replace the file, or upload a new one and modify tez-site to point to this. At the moment, the client is using 0.8, while the AM is

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41377/#review115890 --- Ship it! Ship It! - Siddharth Seth On Jan. 21, 2016, 12:27 a.m

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

Thread uncaughtExceptionHandler / Callable with a ThreadPoolExecutor ? ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 206) <https://reviews.apache.org/r/41377/#comment176450> Is this statement still valid ? If not, please delete or modify accordingly. - Siddharth Seth

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

, and Exception propagation - looks good to me. - Siddharth Seth On Jan. 4, 2016, 11:12 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache

Target version not showing up

Have created an INFRA ticket for this - https://issues.apache.org/jira/browse/INFRA-11276.

Failing tests, precommit test runtimes

There's several tests which are failing consistently. Should we disable these and create blocker jiras to fix and enable thees for the next release. The list of test is at the end of the mail. Also, each pre-commit run seems to be taking over 3 hours. Is this normal ? Having test results come

Re: Failing tests, precommit test runtimes

3 hours. > > Ashutosh is looking I believe, I havent had a chance to take a look myself. > > Szehon > > On Mon, Mar 28, 2016 at 2:28 PM, Siddharth Seth <ss...@apache.org> wrote: > > > There's several tests which are failing consistently. Should we disable > > these a

Re: [discuss] jdk8 support

Option 3 sounds good. I'd ideally like to make JDK8 the minimum requirement soon as well. On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho wrote: > Sounds like a good plan, +1 > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal > wrote: > > > Thanks,

Re: Failing tests, precommit test runtimes

ests till we are able to fix it. Since bug > seems to be in spark runtime, it may take a little while to fix it. > Increased run time for QA till then would be waste of resources. > > Thanks, > Ashutosh > > On Tue, Mar 29, 2016 at 9:06 AM, Siddharth Seth <ss...@apache.org> w

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

> On April 23, 2016, 12:38 a.m., Siddharth Seth wrote: > > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java, > > line 193 > > <https://reviews.apache.org/r/46579/diff/1/?file=1357677#file1357677line193> > > > > Can w

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

mment193941> Can we restrict this configuration parameter to be used within HS2 only ? Any other client (CLI) would have it off by default - and fetch the token from LLAP itself. me - Siddharth Seth On April 22, 2016, 6:47 p.m.

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

P supports multiple instances, which will otherwise end up with different UUIDs. - Siddharth Seth On April 23, 2016, 2:08 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

to make a change before commit - "hs2" as a constant somewhere rather than referring the string literal separately in 2 places, and maybe call it 'hiveserver' in case there's ever an hs3 / hs2 goes away. - Siddharth Seth On April 23, 2016, 1:57 a.m., Sergey Shelu

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

/MiniLlapCluster.java (line 196) <https://reviews.apache.org/r/46597/#comment195150> This can send in a real cluster name, instead of leaving it null ? - Siddharth Seth On April 26, 2016, 12:58 a.m., Sergey Shelukhin

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

UGI instances - 1 per executor, and share those. The allowCache, static lock, compare principal seems unnecessary. - Siddharth Seth On April 27, 2016, 11:01 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically

Re: Review Request 47546: HIVE-13448 LLAP: check ZK acls for ZKSM and fail if they are too permissive

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47546/#review134040 --- Ship it! Ship It! - Siddharth Seth On May 18, 2016, 7:07

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

ould be passed in. The KeySigner could be an interface instead, and SecretManager (extends ZKDelegationTokenSecretManager) can implement this. ACL checks etc are already setup there. There's no requirement to have two independent copies of the ZKSecretManager running in the same daemon.

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote: > > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java, > > line 290 > > <https://reviews.apache.org/r/46956/diff/2/?file=1387298#file1387298line290> > > > > Wha

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote: > > llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java, > > line 134 > > <https://reviews.apache.org/r/46956/diff/2/?file=1387295#file1387295line134> > > > > Can

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

che.org/r/46956/#comment199239> This should explicitly check for "false". Any other value should either default to true or thrown an error. Err on the side of being secure. - Siddharth Seth On May 21, 2016, 12:07 a.m.,

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

issue. - Siddharth Seth On May 20, 2016, 1:49 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

p_owner" end up getting treated as false. ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java <https://reviews.apache.org/r/46956/#comment199458> This parameter can be removed from HiveConf now that it's not used. - Siddharth Seth On May 23

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

a performance perspective, as well as to make sure the threads started by the SecretManager are shutdown. - Siddharth Seth On May 17, 2016, 12:56 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail.

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

- > > (Updated May 17, 2016, 9:58 p.m.) > > > Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit > Kumaraswamy. > > > Repository: hive-git > > > Description > --- > > see JIRA > > > Diffs

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46579/#review133646 --- Ship it! Ship It! - Siddharth Seth On May 17, 2016, 9:58

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

Looks like not sharing the SecretManager could be expensive given the KDC login each time ? - Siddharth Seth On May 5, 2016, 9:55 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To r

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

/LlapSecurityHelper.java (line 95) <https://reviews.apache.org/r/46754/#comment195512> Wondering if this ends up being a little too brittle. HADOOP-13081 itself may break this code. - Siddharth Seth On May 2, 2016, 9:44 p.m., Sergey Shelukhin

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46597/#review131512 --- Ship it! Looks good. - Siddharth Seth On May 2, 2016, 10:01

Re: Review Request 48886: HIVE-14052: Cleanup of structures required when LLAP access from external clients completes

n a FragmentCompletionHandler rather than a Closeable ? ql/src/test/org/apache/hadoop/hive/llap/TestLlapOutputFormat.java (line 52) <https://reviews.apache.org/r/48886/#comment205693> Needs tests. ObjectCacheFactory.removeLlapQueryCache(savedQueryId); - Sid

Re: [VOTE] Drop support for Java7 in master branch

+1 On Mon, Feb 27, 2017 at 8:54 PM, Thejas Nair wrote: > There was a [DISCUSS] thread on the topic of moving to jdk8 for unit tests > [1], and many people also expressed the opinion that we should drop JDK 7 > support in Hive. Public updates by Oracle was stopped on Apr

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

resolving. Follow up jiras. +1, assuming it's been tested locally for regresions and the new functionality. - Siddharth Seth On Sept. 1, 2016, 6:35 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To rep

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

> On Aug. 30, 2016, 10:35 p.m., Siddharth Seth wrote: > > llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java, > > line 538 > > <https://reviews.apache.org/r/51312/diff/1/?file=1481607#file1481607line538> > > >

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51312/#review147548 --- Will look at the updated patch tomorrow. - Siddharth Seth

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

> On Aug. 30, 2016, 10:36 p.m., Siddharth Seth wrote: > > Unrelated to this patch - Any idea why 'uniq' is static/ (private static > > final UUID uniq = UUID.randomUUID();) > > Sergey Shelukhin wrote: > it's daemon identity, so it's global per daemon Why is the

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

> On Aug. 30, 2016, 10:35 p.m., Siddharth Seth wrote: > > llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java, > > line 554 > > <https://reviews.apache.org/r/51312/diff/1/?file=1481607#file1481607line554> > > > >

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

stAffinitySplitLocationProvider.java (line 94) <https://reviews.apache.org/r/51312/#comment214490> This isn't part of the patch uploaded to jira? - Siddharth Seth On Aug. 23, 2016, 1:41 a.m., Sergey Shelukhin wrote: > > --

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

static final UUID uniq = UUID.randomUUID();) - Siddharth Seth On Aug. 23, 2016, 1:41 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache

ptest result structure change (batched unit tests)

HIVE-14540 changes ptest to batch unit tests together (instead of invoking mvn test on individual test classes). As a result, output for multiple tests is a common directory - similar to what happens for batched q tests. To identify the relevant directory, lookup the consoleOuput for the run.

Re: Review Request 51397: HIVE-14536 Unit test code cleanup

a smaller window for conflicts. There's several jiras open to fix either testing infrastructure, or individual tests, or get things working in the IDE. I know Zoltan was driving towards getting tests working via the IDE - I don't know how this affects that effort. - Siddharth Seth On Aug. 25

Re: Review Request 51397: HIVE-14536 Unit test code cleanup

> On Aug. 26, 2016, 2:01 a.m., Siddharth Seth wrote: > > Not a lot here related to the patch itself in this comment. > > Looks like everyone has gotten interested in fixing Hive tests at about the > > same time - and there's a good reason for that :) > > I thin

Re: [DISCUSS] Pre-commit tests before commits

) >- Developer Guide: Adding new unit tests ><https://cwiki.apache.org/confluence/display/Hive/ > DeveloperGuide#DeveloperGuide-Addingnewunittests> > (outdated > instructions) > > -- Lefty > > > On Mon, Oct 24, 2016 at 4:49 PM, Siddharth Seth <ss.

Re: build bot on master

Restarted the ptest server. That normally fixes this error. On Fri, Oct 28, 2016 at 10:33 AM, Eugene Koifman wrote: > Hi, > all test runs are failing with the error below. Does anyone know how to > fix it? > > Thanks, > Eugene > > from

Re: [DISCUSS] Yetus pre-commit tests

Peter In terms of the modules - ignoring the time taken - I would vote for asflicense, author, findbugs, javac, maybe javadoc, wrhitespace. Not sure what checkstyle does, and some form of test4tests is already covered in ptest. This will at least help preventing new issues. Fixing the existing set

Number of retained PreCommit runs

Looks like we retain the last 10 runs at max. With tests running in less than an hour, very often, we end up with situations where test reports are not available? I suspect the 10 run limit is due to the size of the logs generated by each run - close to 6GB (on the ptest server). I think it will

Re: [DISCUSS] Pre-commit tests before commits

directly. Thanks, Sid On Mon, Oct 17, 2016 at 12:07 PM, Sergey Shelukhin <ser...@hortonworks.com> wrote: > +1 > > On 16/10/13, 23:10, "Siddharth Seth" <ss...@apache.org> wrote: > > >There's been a lot of work to make the test runs faster, as well as mor

[DISCUSS] Pre-commit tests before commits

There's been a lot of work to make the test runs faster, as well as more reliable via HIVE-14547, HIVE-13503, and several other jiras. Test runtimes are around the 1 hour mark, and going down. There were a few green pre-commit runs (after years?). At the same time, there's still some flaky tests.

Re: [DISCUSS] Pre-commit tests before commits

> >> 6) Avoid adding more read-only tables to init script as this will > increase > >> the initialization time. > >> > >> 7) If the test case does not require explain plan then avoid it as most > >> failures are explain diffs. > >> > >>

Re: Invitation for Hive committers to become ORC committers

Hi Owen, I'd be interested as well, if not too late. Thanks, Sid On Wed, Jan 4, 2017 at 10:34 AM, Owen O'Malley wrote: > Ferd, I've added you. > > Suneel, I'm sorry, but the offer is limited to current Hive committers. > http://people.apache.org/phonebook.html?unix=hive > >

Re: Green unit test results

There's far too many flaky tests to enforce a clean test run prior to commit. However, there also needs to be a very conscious effort to fix flaky tests - otherwise, we're not going to reach the state where this can be enforced. I'm also not sure whether this is just flaky tests, or whether there

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59325/#review175184 --- On May 17, 2017, 12:30 a.m., Siddharth Seth wrote: > > --

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59325/#review175192 ------- On May 17, 2017, 12:30 a.m., Siddharth Seth wrote: > > --- &g

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

time out and > > be cleaned? > > > > also what if readLock blocks all cleanup attempts? > > Siddharth Seth wrote: > In terms of the callable - was trying to keep it simple. Can try making > use of a single callable. > > Why would a readLock b

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

- Thanks, Siddharth Seth

Re: [Announce] New committer: Vineet Garg

Congratulations Vineet. On Tue, May 9, 2017 at 1:24 PM, Ashutosh Chauhan wrote: > The Project Management Committee (PMC) for Apache Hive has invited Vineet > Garg to become a committer and we are pleased to announce that he has > accepted. > > Welcome, Vineet! > > Thanks,

Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

/llap/daemon/impl/TaskExecutorTestHelpers.java 27c426c ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 868eec7 Diff: https://reviews.apache.org/r/59325/diff/1/ Testing --- Thanks, Siddharth Seth

Re: [Announce] New PMC members

Congratulations! On Tue, May 30, 2017 at 8:55 AM, Sergio Pena wrote: > Thanks all for your comments, and congratulations to all new PMC members > too !!! > > - Sergio > > On Sun, May 28, 2017 at 8:43 PM, Rajesh Balamohan > wrote: > > >

Re: [DISCUSS] Unsustainable situation with ptests

Very nice. There was an effort to get fast and green builds back in 2016. There wasn't any strict "must be a green build" before commit at the time though. Instead jiras were filed and the expectation was that they'd be cited / new ones created pre commit(looking at the jiras now - this was likely

Re: [VOTE] Stricter commit guidelines

+1 On Mon, May 14, 2018 at 10:44 PM, Jesus Camacho Rodriguez < jcama...@apache.org> wrote: > After work has been done to ignore most of the tests that were failing > consistently/intermittently [1], I wanted to start this vote to gather > support from the community to be stricter wrt committing

[jira] [Updated] (HIVE-5522) Move split generation into the AM

[ https://issues.apache.org/jira/browse/HIVE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-5522: - Attachment: HIVE-5522.txt Changes to configure the DAG to generate splits in the AM

[jira] [Updated] (HIVE-5522) Move split generation into the AM

[ https://issues.apache.org/jira/browse/HIVE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-5522: - Attachment: HIVE-5522.3.txt Updated patch to use a different encoder, and make use of some custom

[jira] [Created] (HIVE-5620) Tez job progress printing stops after a specific amount of time

Siddharth Seth created HIVE-5620: Summary: Tez job progress printing stops after a specific amount of time Key: HIVE-5620 URL: https://issues.apache.org/jira/browse/HIVE-5620 Project: Hive

[jira] [Updated] (HIVE-5620) Tez job progress printing stops after a specific amount of time

[ https://issues.apache.org/jira/browse/HIVE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-5620: - Attachment: HIVE-5620.txt Simple patch; prints whenever the status changes or every 3 seconds

[jira] [Updated] (HIVE-5620) Tez job progress printing stops after a specific amount of time

[ https://issues.apache.org/jira/browse/HIVE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-5620: - Status: Patch Available (was: Open) Tez job progress printing stops after a specific amount

[jira] [Created] (HIVE-5639) Allow caching of Orc footers in Tez AM

Siddharth Seth created HIVE-5639: Summary: Allow caching of Orc footers in Tez AM Key: HIVE-5639 URL: https://issues.apache.org/jira/browse/HIVE-5639 Project: Hive Issue Type: Improvement

[jira] [Updated] (HIVE-5639) Allow caching of Orc footers in Tez AM

[ https://issues.apache.org/jira/browse/HIVE-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-5639: - Attachment: HIVE-5639.1.txt Patch adds an option to cache footers. Makes sending footers

[jira] [Commented] (HIVE-5639) Allow caching of Orc footers in Tez AM

[ https://issues.apache.org/jira/browse/HIVE-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803758#comment-13803758 ] Siddharth Seth commented on HIVE-5639: -- bq. I see we are catching Throwable here

[jira] [Commented] (HIVE-5639) Allow caching of Orc footers in Tez AM

[ https://issues.apache.org/jira/browse/HIVE-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13805564#comment-13805564 ] Siddharth Seth commented on HIVE-5639: -- You're right, the current catch Throwable

[jira] [Updated] (HIVE-5639) Allow caching of Orc footers in Tez AM

[ https://issues.apache.org/jira/browse/HIVE-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-5639: - Attachment: HIVE-5639-addendum.txt Addendum patch. Logs non IOExceptions in the thread; also

  1   2   3   4   5   >