[jira] [Updated] (HIVE-13513) cleardanglingscratchdir does not work in some version of HDFS
[
https://issues.apache.org/jira/browse/HIVE-13513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated HIVE-13513:
--
Attachment: HIVE-13513.2.patch
Change the patch so that:
1. The process information is written to a separate file, which is closed so
the content is flushed to HDFS
2. Initialize hive log4j setting, so it will pick hive-log4j.properties to do
the logging
> cleardanglingscratchdir does not work in some version of HDFS
> -
>
> Key: HIVE-13513
> URL: https://issues.apache.org/jira/browse/HIVE-13513
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-13513.1.patch, HIVE-13513.2.patch
>
>
> On some Hadoop version, we keep getting "lease recovery" message at the time
> we check for scratchdir by opening for appending:
> {code}
> Failed to APPEND_FILE xxx for DFSClient_NONMAPREDUCE_785768631_1 on 10.0.0.18
> because lease recovery is in progress. Try again later.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2917)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2677)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2984)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2953)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:655)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:421)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2133)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2131)
> {code}
> and
> {code}
> 16/04/14 04:51:56 ERROR hdfs.DFSClient: Failed to close inode 18963
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[10.0.0.12:30010,DS-b355ac2a-a23a-418a-af9b-4c1b4e26afe8,DISK]],
>
> original=[DatanodeInfoWithStorage[10.0.0.12:30010,DS-b355ac2a-a23a-418a-af9b-4c1b4e26afe8,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:951)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1017)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1165)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:470)
> {code}
> The reason is not clear. However, if we remove hsync from SessionState,
> everything works as expected. Attach patch to remove hsync call for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13342) Improve logging in llap decider and throw exception in case llap mode is all but we cannot run in llap.
[
https://issues.apache.org/jira/browse/HIVE-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244012#comment-15244012
]
Hive QA commented on HIVE-13342:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798605/HIVE-13342.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9982 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_char_simple
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7612/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7612/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7612/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798605 - PreCommit-HIVE-TRUNK-Build
> Improve logging in llap decider and throw exception in case llap mode is all
> but we cannot run in llap.
> ---
>
> Key: HIVE-13342
> URL: https://issues.apache.org/jira/browse/HIVE-13342
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13342.1.patch, HIVE-13342.2.patch,
> HIVE-13342.3.patch, HIVE-13342.4.patch
>
>
> Currently we do not log our decisions with respect to llap. Are we running
> everything in llap mode or only parts of the plan. We need more logging.
> Also, if llap mode is all but for some reason, we cannot run the work in llap
> mode, fail and throw an exception advise the user to change the mode to auto.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13525) HoS hangs when job is empty
[ https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243998#comment-15243998 ] Rui Li commented on HIVE-13525: --- Sorry I didn't notice HIVE-13223 when creating the JIRA. [~szehon] do you think it's a dup? > HoS hangs when job is empty > --- > > Key: HIVE-13525 > URL: https://issues.apache.org/jira/browse/HIVE-13525 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-13525.1.patch > > > Observed in local tests. This should be the cause of HIVE-13402. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13525) HoS hangs when job is empty
[ https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13525: -- Status: Patch Available (was: Open) > HoS hangs when job is empty > --- > > Key: HIVE-13525 > URL: https://issues.apache.org/jira/browse/HIVE-13525 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-13525.1.patch > > > Observed in local tests. This should be the cause of HIVE-13402. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13525) HoS hangs when job is empty
[
https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-13525:
--
Attachment: HIVE-13525.1.patch
I think the reason is that we rely on JobStart/JobEnd events to determine if
the job has finished. However, when the job is empty, there won't be such
events at all. So actually we have to just wait on the future. Please refer to
the comments in {{LocalSparkJobStatus}}.
With the patch, I can run all the disabled tests in HIVE-13402.
{{join_empty.q}} and {{join40.q}} fail (but not hang) which I think is because
of something else. Others run successfully.
Also pinging [~chengxiang li] and [~vanzin] for opinions.
> HoS hangs when job is empty
> ---
>
> Key: HIVE-13525
> URL: https://issues.apache.org/jira/browse/HIVE-13525
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-13525.1.patch
>
>
> Observed in local tests. This should be the cause of HIVE-13402.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13509) HCatalog getSplits should ignore the partition with invalid path
[
https://issues.apache.org/jira/browse/HIVE-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243964#comment-15243964
]
Hive QA commented on HIVE-13509:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798886/HIVE-13509.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 9977 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-cte_4.q-orc_merge5.q-vectorization_limit.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7611/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7611/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7611/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798886 - PreCommit-HIVE-TRUNK-Build
> HCatalog getSplits should ignore the partition with invalid path
>
>
> Key: HIVE-13509
> URL: https://issues.apache.org/jira/browse/HIVE-13509
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13509.1.patch, HIVE-13509.patch
>
>
> It is quite common that there is the discrepancy between partition directory
> and its HMS metadata, simply because the directory could be added/deleted
> externally using hdfs shell command. Technically it should be fixed by MSCK
> and alter table .. add/drop command etc, but sometimes it might not be
> practical especially in a multi-tenant env. This discrepancy does not cause
> any problem to Hive, Hive returns no rows for a partition with an invalid
> (e.g. non-existing) path, but it fails the Pig load with HCatLoader, because
> the HCatBaseInputFormat getSplits throws an error when getting a split for a
> non-existing path. The error message might looks like:
> {code}
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://xyz.com:8020/user/hive/warehouse/xyz/date=2016-01-01/country=BR
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> at
> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:162)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6535) JDBC: provide an async API to execute query and fetch results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243937#comment-15243937 ] Lefty Leverenz commented on HIVE-6535: -- No doc needed in the wiki? > JDBC: provide an async API to execute query and fetch results > - > > Key: HIVE-6535 > URL: https://issues.apache.org/jira/browse/HIVE-6535 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC >Affects Versions: 0.14.0, 1.2.1, 2.0.0 >Reporter: Thejas M Nair >Assignee: Vaibhav Gumashta > Fix For: 2.1.0 > > Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch, > HIVE-6535.4.patch, HIVE-6535.5.patch, HIVE-6535.6.patch > > > The hive jdbc client waits query completion during execute() call. It would > be better to block in the jdbc for completion when the results are being > fetched. > This way the application using hive jdbc driver can do other tasks while > asynchronous query execution is happening, until it needs to fetch the result > set. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13522) regexp_extract.q hangs on master
[ https://issues.apache.org/jira/browse/HIVE-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243935#comment-15243935 ] Owen O'Malley commented on HIVE-13522: -- Since Matt reverted the change for HIVE-12159, and I've updated that patch with the fix to this (and reverting this patch), we should probably close this as a duplicate. > regexp_extract.q hangs on master > > > Key: HIVE-13522 > URL: https://issues.apache.org/jira/browse/HIVE-13522 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Blocker > Attachments: HIVE-13522.patch, jstack_regexp_extract.txt > > > Disable to unblock Hive QA runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HIVE-12159: - Attachment: HIVE-12159.patch Ok, LLAP because it is going under the hood to access TreeReaders directly, needs to call ensureSize on the ColumnVector. You can see the delta for this patch on the pull request. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch, HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options
[ https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243911#comment-15243911 ] Eugene Koifman commented on HIVE-11793: --- [~wzheng] 1. I'm not sure it's OK to just change method signatures on HiveMetaStoreClient. I think this is considered public API. Same for IMetaStoreClient. 2. TxnHandler.showLocks(): if dbName == null, should it add "hl_db=default" in the WHERE? 3. dbtxnmgr_showlocks.q: it seems a bit limited since there are no locks in the system. TestDbTxnManager2.java has some examples how you can create some locks for testing. patch in HIVE-13395 has more/better examples also in TestDbTxnManager2. This is not critical > SHOW LOCKS with DbTxnManager ignores filter options > --- > > Key: HIVE-11793 > URL: https://issues.apache.org/jira/browse/HIVE-11793 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch > > > https://cwiki.apache.org/confluence/display/Hive/Locking and > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks > list various options that can be used with SHOW LOCKS, e.g. > When ACID is enabled, all these options are ignored and a full list is > returned. > (also only ext lock id is shown, int lock id is not). > see DDLTask.showLocks() and TxnHandler.showLocks() > requires extending ShowLocksRequest which is a Thrift object -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Dholakia updated HIVE-12049:
--
Attachment: HIVE-12049.19.patch
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.3.patch, HIVE-12049.4.patch, HIVE-12049.5.patch,
> HIVE-12049.6.patch, HIVE-12049.7.patch, HIVE-12049.9.patch,
> new-driver-profiles.png, old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13501) Invoke failure hooks if query fails on exception
[ https://issues.apache.org/jira/browse/HIVE-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13501: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Integrated into trunk. Thanks Szehon for the review. > Invoke failure hooks if query fails on exception > > > Key: HIVE-13501 > URL: https://issues.apache.org/jira/browse/HIVE-13501 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-13501.1.patch, HIVE-13501.2.patch > > > When a query fails on some exception, failure hooks are not called currently. > It's better to invoke such hooks so that we know the query is failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243901#comment-15243901 ] Eugene Koifman commented on HIVE-13395: --- [~alangates] could you review please? > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.6.patch, HIVE-13395.7.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Here is one simple example why we need Write-Set tracking for multi-statement > txns > Consider transactions T ~1~ and T ~2~: > T ~1~: r ~1~\[x] -> w ~1~\[y] -> c ~1~ > T ~2~: w ~2~\[x] -> w ~2~\[y] -> c ~2~ > Suppose the order of operations is r ~1~\[x] w ~2~\[x] then
[jira] [Commented] (HIVE-12637) make retryable SQLExceptions in TxnHandler configurable
[
https://issues.apache.org/jira/browse/HIVE-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243890#comment-15243890
]
Eugene Koifman commented on HIVE-12637:
---
[~wzheng]
1. For the description of HIVE_TXN_RETRYABLE_SQLEX_REGEX I would add that the
string that the regex will check is "ex.getMessage() + "(SQLState=" +
ex.getSQLState() + ",ErrorCode=" + ex.getErrorCode() + ")"" where "ex" is a
SQLException. I think this would make it more intuitive for end users.
2. testRetryableRegex() could you add a test where the error msg includes a ","?
> make retryable SQLExceptions in TxnHandler configurable
> ---
>
> Key: HIVE-12637
> URL: https://issues.apache.org/jira/browse/HIVE-12637
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Attachments: HIVE-12637.1.patch
>
>
> same for CompactionTxnHandler
> would be convenient if the user could specify some RegEx (perhaps by db type)
> which will tell TxnHandler.checkRetryable() that this is should be retried.
> The regex should probably apply to String produced by
> {noformat}
> private static String getMessage(SQLException ex) {
> return ex.getMessage() + "(SQLState=" + ex.getSQLState() + ",ErrorCode="
> + ex.getErrorCode() + ")";
> }
> {noformat}
> This make it flexible.
> See if we need to add Db type (and possibly version) of the DB being used.
> With 5 different DBs supported this gives control end users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13424) Refactoring the code to pass a QueryState object rather than HiveConf object
[
https://issues.apache.org/jira/browse/HIVE-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243868#comment-15243868
]
Hive QA commented on HIVE-13424:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798565/HIVE-13424.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7610/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7610/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7610/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO]
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-service-rpc ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @
hive-service-rpc ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.1.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.1.0-SNAPSHOT/hive-service-rpc-2.1.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/service-rpc/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.1.0-SNAPSHOT/hive-service-rpc-2.1.0-SNAPSHOT.pom
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.1.0-SNAPSHOT-tests.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.1.0-SNAPSHOT/hive-service-rpc-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO]
[INFO] Building Spark Remote Client 2.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ spark-client ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/spark-client/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/spark-client (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
spark-client ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-client
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
spark-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ spark-client ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ spark-client
---
[INFO] Compiling 28 source files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java:
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java
uses or overrides a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java:
Some input files use unchecked or unsafe operations.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcDispatcher.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
spark-client ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ spark-client ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[copy] Copying 15 files to
/data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf
[jira] [Commented] (HIVE-13522) regexp_extract.q hangs on master
[ https://issues.apache.org/jira/browse/HIVE-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243865#comment-15243865 ] Owen O'Malley commented on HIVE-13522: -- Ok, more details, the real problem is llap_partitioned.q. Taking that test out of the mix makes the batch pass and it also fails when run on its own. It is getting an array out of bounds exception in the vectorized ORC reader and I'm working on tracking down how/why. > regexp_extract.q hangs on master > > > Key: HIVE-13522 > URL: https://issues.apache.org/jira/browse/HIVE-13522 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Blocker > Attachments: HIVE-13522.patch, jstack_regexp_extract.txt > > > Disable to unblock Hive QA runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13529) Move around some of the classes created during llap branch work
[ https://issues.apache.org/jira/browse/HIVE-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere resolved HIVE-13529. --- Resolution: Fixed Committed to llap branch > Move around some of the classes created during llap branch work > --- > > Key: HIVE-13529 > URL: https://issues.apache.org/jira/browse/HIVE-13529 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: llap > > Attachments: HIVE-13529.1.patch > > > Try to move around some of the classes created, to get the dependencies to > work a little better for clients. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13529) Move around some of the classes created during llap branch work
[ https://issues.apache.org/jira/browse/HIVE-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13529: -- Attachment: HIVE-13529.1.patch Forgot to move the LlapInputSplit from the jdbc/ package. I'm hoping to get rid of these wrapper LlapInputFormat/LlapInputSplit classes in later work. > Move around some of the classes created during llap branch work > --- > > Key: HIVE-13529 > URL: https://issues.apache.org/jira/browse/HIVE-13529 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: llap > > Attachments: HIVE-13529.1.patch > > > Try to move around some of the classes created, to get the dependencies to > work a little better for clients. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12634) Add command to kill an ACID transacton
[
https://issues.apache.org/jira/browse/HIVE-12634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243859#comment-15243859
]
Eugene Koifman commented on HIVE-12634:
---
1. did you figure out why there are so many changes in generated code and if
it's a reason for concern?
2. I think that the new _TxnHandler.abortTxns(AbortTxnsRequest rqst)_ should
best effort, i.e. if the parameter list is 10 txn ids but for some reason it
only aborted 7 that should be ok. _abortTxns(Connection dbConn, List
txnids, long max_heartbeat)_ is too restrictive as currently implemented. It
should only bail when it didn't find all requested txn ids in some case.
3. HIVE-12439 introduced a bad bug in _abortTxns(Connection dbConn, List
txnids, long max_heartbeat)_. It changed how
{noformat}
updateCnt = stmt.executeUpdate(query);
if (updateCnt < txnids.size()) {
}
{noformat}
works. Now the 'if' statement will be true every time
_buildQueryWithINClause()_ produces > 1 query.
> Add command to kill an ACID transacton
> --
>
> Key: HIVE-12634
> URL: https://issues.apache.org/jira/browse/HIVE-12634
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
> Attachments: HIVE-12634.1.patch, HIVE-12634.2.patch
>
>
> Should add a CLI command to abort a (runaway) transaction.
> This should clean up all state related to this txn.
> The initiator of this (if still alive) will get an error trying to
> heartbeat/commit, i.e. will become aware that the txn is dead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13529) Move around some of the classes created during llap branch work
[ https://issues.apache.org/jira/browse/HIVE-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13529: -- Attachment: (was: HIVE-13529.1.patch) > Move around some of the classes created during llap branch work > --- > > Key: HIVE-13529 > URL: https://issues.apache.org/jira/browse/HIVE-13529 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: llap > > > Try to move around some of the classes created, to get the dependencies to > work a little better for clients. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13440) remove hiveserver1 scripts under bin/ext/
[
https://issues.apache.org/jira/browse/HIVE-13440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243858#comment-15243858
]
Hive QA commented on HIVE-13440:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798560/HIVE-13440.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9979 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7609/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7609/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7609/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798560 - PreCommit-HIVE-TRUNK-Build
> remove hiveserver1 scripts under bin/ext/
> -
>
> Key: HIVE-13440
> URL: https://issues.apache.org/jira/browse/HIVE-13440
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Thejas M Nair
>Assignee: Balint Molnar
> Labels: newbie, trivial
> Attachments: HIVE-13440.patch
>
>
> HIVE-6977 deleted hiveserver1, however the scripts remain under bin/ext/-
> ls bin/ext/hiveserver.*
> bin/ext/hiveserver.cmd bin/ext/hiveserver.sh
> The should be removed as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13530) Hive on Spark throws Kryo exception in some cases
[
https://issues.apache.org/jira/browse/HIVE-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-13530:
-
Status: Patch Available (was: Open)
> Hive on Spark throws Kryo exception in some cases
> -
>
> Key: HIVE-13530
> URL: https://issues.apache.org/jira/browse/HIVE-13530
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-13530.patch
>
>
> After recent changes, Hive on Spark throws KryoException:
> {noformat}
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: ERROR : Failed to execute spark
> task, with exception 'java.lang.Exception(Failed to submit Spark work, please
> retry later)'
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: java.lang.Exception: Failed to
> submit Spark work, please retry later
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.execute(RemoteHiveSparkClient.java:174)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.submit(SparkSessionImpl.java:71)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:103)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1769)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1526)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1305)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1114)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1107)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.security.AccessController.doPrivileged(Native Method)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> javax.security.auth.Subject.doAs(Subject.java:415)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.lang.Thread.run(Thread.java:745)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Caused by:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.util.ConcurrentModificationException
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Serialization trace:
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classes
> (sun.misc.Launcher$AppClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classloader
> (java.security.ProtectionDomain)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: context
> (java.security.AccessControlContext)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: acc (java.net.URLClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classLoader
> (org.apache.hadoop.hive.conf.HiveConf)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: conf
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: metrics
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$CodahaleMetricsScope)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: openScopes
> (org.apache.hadoop.hive.ql.log.PerfLogger)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: perfLogger
> (org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: childOperators
> (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> 16/04/14
[jira] [Updated] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13097: -- Target Version/s: 2.1.0 Status: Patch Available (was: Open) > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13097: -- Attachment: HIVE-13097.01.patch Combined patch - including HIVE-13184, HIVE-13431 and HIVE-13433 for jenkins. > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.01.patch, HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13097) [Umbrella] Changes dependent on Tez 0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13097: -- Attachment: HIVE-13097.versonChange.patch Trivial patch to change the version. > [Umbrella] Changes dependent on Tez 0.8.3 > - > > Key: HIVE-13097 > URL: https://issues.apache.org/jira/browse/HIVE-13097 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth > Attachments: HIVE-13097.versonChange.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13433) Fixes for additional incompatible changes in tez-0.8.3
[ https://issues.apache.org/jira/browse/HIVE-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13433: -- Attachment: HIVE-13433.02.patch Rebased patch on top of HIVE-13184, and some test fixes. > Fixes for additional incompatible changes in tez-0.8.3 > -- > > Key: HIVE-13433 > URL: https://issues.apache.org/jira/browse/HIVE-13433 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13433.02.patch, HIVE-13433.1.txt > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13498) cleardanglingscratchdir does not work if scratchdir is not on defaultFs
[ https://issues.apache.org/jira/browse/HIVE-13498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-13498: -- Target Version/s: 1.3.0, 2.1.0 > cleardanglingscratchdir does not work if scratchdir is not on defaultFs > --- > > Key: HIVE-13498 > URL: https://issues.apache.org/jira/browse/HIVE-13498 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13498.1.patch > > > The cleardanglingscratchdir utility need a fix to make it work if scratchdir > is not on defaultFs, such as on Azure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13498) cleardanglingscratchdir does not work if scratchdir is not on defaultFs
[ https://issues.apache.org/jira/browse/HIVE-13498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-13498: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Test failures are not related. Patch pushed to master and branch-1. > cleardanglingscratchdir does not work if scratchdir is not on defaultFs > --- > > Key: HIVE-13498 > URL: https://issues.apache.org/jira/browse/HIVE-13498 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13498.1.patch > > > The cleardanglingscratchdir utility need a fix to make it work if scratchdir > is not on defaultFs, such as on Azure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13509) HCatalog getSplits should ignore the partition with invalid path
[
https://issues.apache.org/jira/browse/HIVE-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243789#comment-15243789
]
Mithun Radhakrishnan commented on HIVE-13509:
-
I'm stuck on production-support, at the moment. I'd review this on Monday.
Sorry for the delay.
> HCatalog getSplits should ignore the partition with invalid path
>
>
> Key: HIVE-13509
> URL: https://issues.apache.org/jira/browse/HIVE-13509
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13509.1.patch, HIVE-13509.patch
>
>
> It is quite common that there is the discrepancy between partition directory
> and its HMS metadata, simply because the directory could be added/deleted
> externally using hdfs shell command. Technically it should be fixed by MSCK
> and alter table .. add/drop command etc, but sometimes it might not be
> practical especially in a multi-tenant env. This discrepancy does not cause
> any problem to Hive, Hive returns no rows for a partition with an invalid
> (e.g. non-existing) path, but it fails the Pig load with HCatLoader, because
> the HCatBaseInputFormat getSplits throws an error when getting a split for a
> non-existing path. The error message might looks like:
> {code}
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://xyz.com:8020/user/hive/warehouse/xyz/date=2016-01-01/country=BR
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> at
> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:162)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-6535) JDBC: provide an async API to execute query and fetch results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-6535: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 Release Note: Starts the query execution asynchronously on the server, and immediately returns to the client. The client subsequently blocks on ResultSet#next or Statement#getUpdateCount, depending on the query type. Users should call ResultSet.next or Statement#getUpdateCount (depending on whether query returns results) to ensure that query completes successfully. Calling another execute method, or close before query completion would result in the async query getting killed if it is not already finished. Note: This method is an API for limited usage outside of Hive by applications like Apache Ambari, although it is not part of the interface java.sql.Statement. Target Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. Thanks [~thejas] for reviewing. > JDBC: provide an async API to execute query and fetch results > - > > Key: HIVE-6535 > URL: https://issues.apache.org/jira/browse/HIVE-6535 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC >Affects Versions: 0.14.0, 1.2.1, 2.0.0 >Reporter: Thejas M Nair >Assignee: Vaibhav Gumashta > Fix For: 2.1.0 > > Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch, > HIVE-6535.4.patch, HIVE-6535.5.patch, HIVE-6535.6.patch > > > The hive jdbc client waits query completion during execute() call. It would > be better to block in the jdbc for completion when the results are being > fetched. > This way the application using hive jdbc driver can do other tasks while > asynchronous query execution is happening, until it needs to fetch the result > set. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6535) JDBC: provide an async API to execute query and fetch results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-6535: --- Summary: JDBC: provide an async API to execute query and fetch results (was: JDBC: async wait should happen during fetch for results) > JDBC: provide an async API to execute query and fetch results > - > > Key: HIVE-6535 > URL: https://issues.apache.org/jira/browse/HIVE-6535 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC >Affects Versions: 0.14.0, 1.2.1, 2.0.0 >Reporter: Thejas M Nair >Assignee: Vaibhav Gumashta > Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch, > HIVE-6535.4.patch, HIVE-6535.5.patch, HIVE-6535.6.patch > > > The hive jdbc client waits query completion during execute() call. It would > be better to block in the jdbc for completion when the results are being > fetched. > This way the application using hive jdbc driver can do other tasks while > asynchronous query execution is happening, until it needs to fetch the result > set. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6535) JDBC: async wait should happen during fetch for results
[ https://issues.apache.org/jira/browse/HIVE-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243756#comment-15243756 ] Vaibhav Gumashta commented on HIVE-6535: Looks like TestJdbcWithMiniHS2 was disabled when the QA run for this patch happened. I ran it on latest master and it runs fine. Rest of the failures are unrelated. I'll commit shortly. > JDBC: async wait should happen during fetch for results > --- > > Key: HIVE-6535 > URL: https://issues.apache.org/jira/browse/HIVE-6535 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC >Affects Versions: 0.14.0, 1.2.1, 2.0.0 >Reporter: Thejas M Nair >Assignee: Vaibhav Gumashta > Attachments: HIVE-6535.1.patch, HIVE-6535.2.patch, HIVE-6535.3.patch, > HIVE-6535.4.patch, HIVE-6535.5.patch, HIVE-6535.6.patch > > > The hive jdbc client waits query completion during execute() call. It would > be better to block in the jdbc for completion when the results are being > fetched. > This way the application using hive jdbc driver can do other tasks while > asynchronous query execution is happening, until it needs to fetch the result > set. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12959) LLAP: Add task scheduler timeout when no nodes are alive
[ https://issues.apache.org/jira/browse/HIVE-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12959: - Status: Patch Available (was: Open) > LLAP: Add task scheduler timeout when no nodes are alive > > > Key: HIVE-12959 > URL: https://issues.apache.org/jira/browse/HIVE-12959 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12959.1.patch, HIVE-12959.2.patch, > HIVE-12959.3.patch, HIVE-12959.5.patch, HIVE-12959.6.patch, HIVE-12959.6.patch > > > When there are no llap daemons running task scheduler should have a timeout > to fail the query instead of waiting forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12959) LLAP: Add task scheduler timeout when no nodes are alive
[ https://issues.apache.org/jira/browse/HIVE-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-12959: - Attachment: HIVE-12959.6.patch In this patch, using 0.8.3-SNAPSHOT version of tez just for precommit test run. > LLAP: Add task scheduler timeout when no nodes are alive > > > Key: HIVE-12959 > URL: https://issues.apache.org/jira/browse/HIVE-12959 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-12959.1.patch, HIVE-12959.2.patch, > HIVE-12959.3.patch, HIVE-12959.5.patch, HIVE-12959.6.patch, HIVE-12959.6.patch > > > When there are no llap daemons running task scheduler should have a timeout > to fail the query instead of waiting forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13469) LLAP: Support delayed scheduling for locality
[ https://issues.apache.org/jira/browse/HIVE-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243750#comment-15243750 ] Prasanth Jayachandran commented on HIVE-13469: -- Mostly looks good to me. Left very minor comments in RB. Otherwise, +1 > LLAP: Support delayed scheduling for locality > - > > Key: HIVE-13469 > URL: https://issues.apache.org/jira/browse/HIVE-13469 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13469.01.patch > > > LLAP currently supports forcing locality. Change this to support a time based > delay for locality as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12019) Create unit test for HIVE-10732
[ https://issues.apache.org/jira/browse/HIVE-12019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-12019: -- Attachment: HIVE-12019.2.patch Rebasing with master > Create unit test for HIVE-10732 > --- > > Key: HIVE-12019 > URL: https://issues.apache.org/jira/browse/HIVE-12019 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-12019.1.patch, HIVE-12019.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13501) Invoke failure hooks if query fails on exception
[
https://issues.apache.org/jira/browse/HIVE-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243728#comment-15243728
]
Hive QA commented on HIVE-13501:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798982/HIVE-13501.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 9979 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7608/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7608/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7608/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798982 - PreCommit-HIVE-TRUNK-Build
> Invoke failure hooks if query fails on exception
>
>
> Key: HIVE-13501
> URL: https://issues.apache.org/jira/browse/HIVE-13501
> Project: Hive
> Issue Type: Improvement
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: HIVE-13501.1.patch, HIVE-13501.2.patch
>
>
> When a query fails on some exception, failure hooks are not called currently.
> It's better to invoke such hooks so that we know the query is failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13530) Hive on Spark throws Kryo exception in some cases
[
https://issues.apache.org/jira/browse/HIVE-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-13530:
-
Attachment: HIVE-13530.patch
I think this is my fault from HIVE-13410. The new field is bringing in all of
codahale.
[~jxiang] do you think it makes sense? Thanks.
> Hive on Spark throws Kryo exception in some cases
> -
>
> Key: HIVE-13530
> URL: https://issues.apache.org/jira/browse/HIVE-13530
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Szehon Ho
> Attachments: HIVE-13530.patch
>
>
> After recent changes, Hive on Spark throws KryoException:
> {noformat}
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: ERROR : Failed to execute spark
> task, with exception 'java.lang.Exception(Failed to submit Spark work, please
> retry later)'
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: java.lang.Exception: Failed to
> submit Spark work, please retry later
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.execute(RemoteHiveSparkClient.java:174)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.submit(SparkSessionImpl.java:71)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:103)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1769)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1526)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1305)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1114)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1107)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.security.AccessController.doPrivileged(Native Method)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> javax.security.auth.Subject.doAs(Subject.java:415)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.lang.Thread.run(Thread.java:745)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Caused by:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.util.ConcurrentModificationException
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Serialization trace:
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classes
> (sun.misc.Launcher$AppClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classloader
> (java.security.ProtectionDomain)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: context
> (java.security.AccessControlContext)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: acc (java.net.URLClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classLoader
> (org.apache.hadoop.hive.conf.HiveConf)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: conf
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: metrics
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$CodahaleMetricsScope)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: openScopes
> (org.apache.hadoop.hive.ql.log.PerfLogger)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: perfLogger
> (org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator)
> 16/04/14 21:53:24 INFO
[jira] [Assigned] (HIVE-13530) Hive on Spark throws Kryo exception in some cases
[
https://issues.apache.org/jira/browse/HIVE-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned HIVE-13530:
Assignee: Szehon Ho
> Hive on Spark throws Kryo exception in some cases
> -
>
> Key: HIVE-13530
> URL: https://issues.apache.org/jira/browse/HIVE-13530
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-13530.patch
>
>
> After recent changes, Hive on Spark throws KryoException:
> {noformat}
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: ERROR : Failed to execute spark
> task, with exception 'java.lang.Exception(Failed to submit Spark work, please
> retry later)'
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: java.lang.Exception: Failed to
> submit Spark work, please retry later
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.execute(RemoteHiveSparkClient.java:174)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.submit(SparkSessionImpl.java:71)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:103)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1769)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1526)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1305)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1114)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1107)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.security.AccessController.doPrivileged(Native Method)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> javax.security.auth.Subject.doAs(Subject.java:415)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.lang.Thread.run(Thread.java:745)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Caused by:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.util.ConcurrentModificationException
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Serialization trace:
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classes
> (sun.misc.Launcher$AppClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classloader
> (java.security.ProtectionDomain)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: context
> (java.security.AccessControlContext)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: acc (java.net.URLClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classLoader
> (org.apache.hadoop.hive.conf.HiveConf)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: conf
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: metrics
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$CodahaleMetricsScope)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: openScopes
> (org.apache.hadoop.hive.ql.log.PerfLogger)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: perfLogger
> (org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: childOperators
> (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> 16/04/14 21:53:24
[jira] [Commented] (HIVE-13505) Skip running TestDummy where possibe during precommit builds
[ https://issues.apache.org/jira/browse/HIVE-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243671#comment-15243671 ] Szehon Ho commented on HIVE-13505: -- OK, sure , i havent had a chance to look at it. Can you try and work with [~spena] to coordinate with restarting it? > Skip running TestDummy where possibe during precommit builds > > > Key: HIVE-13505 > URL: https://issues.apache.org/jira/browse/HIVE-13505 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13505.01.patch > > > On the main Hive build - this does nothing. There are some tests named > TestDummy under qtests - I'm not sure they do anything useful though. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13469) LLAP: Support delayed scheduling for locality
[ https://issues.apache.org/jira/browse/HIVE-13469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13469: -- Status: Patch Available (was: Open) > LLAP: Support delayed scheduling for locality > - > > Key: HIVE-13469 > URL: https://issues.apache.org/jira/browse/HIVE-13469 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13469.01.patch > > > LLAP currently supports forcing locality. Change this to support a time based > delay for locality as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13530) Hive on Spark throws Kryo exception in some cases
[
https://issues.apache.org/jira/browse/HIVE-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-13530:
-
Component/s: Spark
> Hive on Spark throws Kryo exception in some cases
> -
>
> Key: HIVE-13530
> URL: https://issues.apache.org/jira/browse/HIVE-13530
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Szehon Ho
>
> After recent changes, Hive on Spark throws KryoException:
> {noformat}
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: ERROR : Failed to execute spark
> task, with exception 'java.lang.Exception(Failed to submit Spark work, please
> retry later)'
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: java.lang.Exception: Failed to
> submit Spark work, please retry later
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.execute(RemoteHiveSparkClient.java:174)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.submit(SparkSessionImpl.java:71)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:103)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1769)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1526)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1305)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1114)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1107)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:178)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:72)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:232)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.security.AccessController.doPrivileged(Native Method)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> javax.security.auth.Subject.doAs(Subject.java:415)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:245)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: at
> java.lang.Thread.run(Thread.java:745)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Caused by:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.util.ConcurrentModificationException
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: Serialization trace:
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classes
> (sun.misc.Launcher$AppClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classloader
> (java.security.ProtectionDomain)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: context
> (java.security.AccessControlContext)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: acc (java.net.URLClassLoader)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: classLoader
> (org.apache.hadoop.hive.conf.HiveConf)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: conf
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: metrics
> (org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$CodahaleMetricsScope)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: openScopes
> (org.apache.hadoop.hive.ql.log.PerfLogger)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: perfLogger
> (org.apache.hadoop.hive.ql.exec.SparkHashTableSinkOperator)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: childOperators
> (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> 16/04/14 21:53:24 INFO hiveserver2.DDLTest: aliasToWork
> (org.apache.hadoop.hive.ql.plan.MapWork)
>
[jira] [Updated] (HIVE-13529) Move around some of the classes created during llap branch work
[ https://issues.apache.org/jira/browse/HIVE-13529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13529: -- Attachment: HIVE-13529.1.patch Move client-related classes either to llap-client or to the new llap-ext-client package > Move around some of the classes created during llap branch work > --- > > Key: HIVE-13529 > URL: https://issues.apache.org/jira/browse/HIVE-13529 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: llap > > Attachments: HIVE-13529.1.patch > > > Try to move around some of the classes created, to get the dependencies to > work a little better for clients. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13498) cleardanglingscratchdir does not work if scratchdir is not on defaultFs
[
https://issues.apache.org/jira/browse/HIVE-13498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243527#comment-15243527
]
Hive QA commented on HIVE-13498:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798442/HIVE-13498.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9952 tests executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
TestMiniTezCliDriver-dynpart_sort_optimization2.q-cte_mat_1.q-tez_bmj_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7607/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7607/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7607/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798442 - PreCommit-HIVE-TRUNK-Build
> cleardanglingscratchdir does not work if scratchdir is not on defaultFs
> ---
>
> Key: HIVE-13498
> URL: https://issues.apache.org/jira/browse/HIVE-13498
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-13498.1.patch
>
>
> The cleardanglingscratchdir utility need a fix to make it work if scratchdir
> is not on defaultFs, such as on Azure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13527) Using deprecated APIs in HBase client causes zookeeper connection leaks.
[
https://issues.apache.org/jira/browse/HIVE-13527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-13527:
-
Status: Patch Available (was: Open)
> Using deprecated APIs in HBase client causes zookeeper connection leaks.
>
>
> Key: HIVE-13527
> URL: https://issues.apache.org/jira/browse/HIVE-13527
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13527.patch
>
>
> When running queries against hbase-backed hive tables, the following log
> messages are seen in the HS2 log.
> {code}
> 2016-04-11 07:25:23,657 WARN
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: You are using an
> HTable instance that relies on an HBase-managed Connection. This is usually
> due to directly creating an HTable, which is deprecated. Instead, you should
> create a Connection object and then request a Table instance from it. If you
> don't need the Table instance for your own use, you should instead use the
> TableInputFormatBase.initalizeTable method directly.
> 2016-04-11 07:25:23,658 INFO
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: Creating an
> additional unmanaged connection because user provided one can't be used for
> administrative actions. We'll close it when we close out the table.
> {code}
> In a HS2 log file, there are 1366 zookeeper connections established but only
> a small fraction of them were closed. So lsof would show 1300+ open TCP
> connections to Zookeeper.
> grep "org.apache.zookeeper.ClientCnxn: Session establishment complete on
> server" * |wc -l
> 1366
> grep "INFO org.apache.zookeeper.ZooKeeper: Session:" * |grep closed |wc -l
> 54
> According to the comments in TableInputFormatBase, the recommended means for
> subclasses like HiveHBaseTableInputFormat is to call initializeTable()
> instead of setHTable() that it currently uses.
> "
> Subclasses MUST ensure initializeTable(Connection, TableName) is called for
> an instance to function properly. Each of the entry points to this class used
> by the MapReduce framework, {@link #createRecordReader(InputSplit,
> TaskAttemptContext)} and {@link #getSplits(JobContext)}, will call {@link
> #initialize(JobContext)} as a convenient centralized location to handle
> retrieving the necessary configuration information. If your subclass
> overrides either of these methods, either call the parent version or call
> initialize yourself.
> "
> Currently setHTable() also creates an additional Admin connection, even
> though it is not needed.
> So the use of deprecated APIs are to be replaced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12159) Create vectorized readers for the complex types
[ https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243462#comment-15243462 ] Thejas M Nair commented on HIVE-12159: -- [~owen.omalley] [~mmccline] Are you able to identify a fix for this ? If not, I think we should temporarily revert the change. We don't know if this issue going to be isolated to tests. > Create vectorized readers for the complex types > --- > > Key: HIVE-12159 > URL: https://issues.apache.org/jira/browse/HIVE-12159 > Project: Hive > Issue Type: Sub-task >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-12159.patch, HIVE-12159.patch, HIVE-12159.patch, > HIVE-12159.patch > > > We need vectorized readers for the complex types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13527) Using deprecated APIs in HBase client causes zookeeper connection leaks.
[
https://issues.apache.org/jira/browse/HIVE-13527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-13527:
-
Attachment: HIVE-13527.patch
Attaching a patch that removes the usage of setHTable() from the
TableInputFormatBase.
> Using deprecated APIs in HBase client causes zookeeper connection leaks.
>
>
> Key: HIVE-13527
> URL: https://issues.apache.org/jira/browse/HIVE-13527
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-13527.patch
>
>
> When running queries against hbase-backed hive tables, the following log
> messages are seen in the HS2 log.
> {code}
> 2016-04-11 07:25:23,657 WARN
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: You are using an
> HTable instance that relies on an HBase-managed Connection. This is usually
> due to directly creating an HTable, which is deprecated. Instead, you should
> create a Connection object and then request a Table instance from it. If you
> don't need the Table instance for your own use, you should instead use the
> TableInputFormatBase.initalizeTable method directly.
> 2016-04-11 07:25:23,658 INFO
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase: Creating an
> additional unmanaged connection because user provided one can't be used for
> administrative actions. We'll close it when we close out the table.
> {code}
> In a HS2 log file, there are 1366 zookeeper connections established but only
> a small fraction of them were closed. So lsof would show 1300+ open TCP
> connections to Zookeeper.
> grep "org.apache.zookeeper.ClientCnxn: Session establishment complete on
> server" * |wc -l
> 1366
> grep "INFO org.apache.zookeeper.ZooKeeper: Session:" * |grep closed |wc -l
> 54
> According to the comments in TableInputFormatBase, the recommended means for
> subclasses like HiveHBaseTableInputFormat is to call initializeTable()
> instead of setHTable() that it currently uses.
> "
> Subclasses MUST ensure initializeTable(Connection, TableName) is called for
> an instance to function properly. Each of the entry points to this class used
> by the MapReduce framework, {@link #createRecordReader(InputSplit,
> TaskAttemptContext)} and {@link #getSplits(JobContext)}, will call {@link
> #initialize(JobContext)} as a convenient centralized location to handle
> retrieving the necessary configuration information. If your subclass
> overrides either of these methods, either call the parent version or call
> initialize yourself.
> "
> Currently setHTable() also creates an additional Admin connection, even
> though it is not needed.
> So the use of deprecated APIs are to be replaced.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13499) TestJdbcWithMiniHS2 is hanging
[ https://issues.apache.org/jira/browse/HIVE-13499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243446#comment-15243446 ] Thejas M Nair commented on HIVE-13499: -- Reverted the temporary change to disable the test as the change that was causing this issue has been reverted. > TestJdbcWithMiniHS2 is hanging > -- > > Key: HIVE-13499 > URL: https://issues.apache.org/jira/browse/HIVE-13499 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Thejas M Nair >Assignee: Aihua Xu > Attachments: HIVE-13499.temp.patch, HIVE-13499.temp2.patch > > > After HIVE-13149 went in , TestJdbcWithMiniHS2. -testConcurrentStatements- > has been hanging, causing delays in the unit test run. > Update : looks like it might be affecting more than one test in this class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13499) TestJdbcWithMiniHS2 is hanging
[ https://issues.apache.org/jira/browse/HIVE-13499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair resolved HIVE-13499. -- Resolution: Not A Problem resolving as not a problem (anymore). > TestJdbcWithMiniHS2 is hanging > -- > > Key: HIVE-13499 > URL: https://issues.apache.org/jira/browse/HIVE-13499 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Thejas M Nair >Assignee: Aihua Xu > Attachments: HIVE-13499.temp.patch, HIVE-13499.temp2.patch > > > After HIVE-13149 went in , TestJdbcWithMiniHS2. -testConcurrentStatements- > has been hanging, causing delays in the unit test run. > Update : looks like it might be affecting more than one test in this class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13509) HCatalog getSplits should ignore the partition with invalid path
[
https://issues.apache.org/jira/browse/HIVE-13509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243410#comment-15243410
]
Chaoyu Tang commented on HIVE-13509:
[~mithun] Could you take a look at the patch to see if it is good to you?
> HCatalog getSplits should ignore the partition with invalid path
>
>
> Key: HIVE-13509
> URL: https://issues.apache.org/jira/browse/HIVE-13509
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-13509.1.patch, HIVE-13509.patch
>
>
> It is quite common that there is the discrepancy between partition directory
> and its HMS metadata, simply because the directory could be added/deleted
> externally using hdfs shell command. Technically it should be fixed by MSCK
> and alter table .. add/drop command etc, but sometimes it might not be
> practical especially in a multi-tenant env. This discrepancy does not cause
> any problem to Hive, Hive returns no rows for a partition with an invalid
> (e.g. non-existing) path, but it fails the Pig load with HCatLoader, because
> the HCatBaseInputFormat getSplits throws an error when getting a split for a
> non-existing path. The error message might looks like:
> {code}
> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does
> not exist:
> hdfs://xyz.com:8020/user/hive/warehouse/xyz/date=2016-01-01/country=BR
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
> at
> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:162)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13525) HoS hangs when job is empty
[ https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243271#comment-15243271 ] Szehon Ho commented on HIVE-13525: -- Yea it looks related to HIVE-13223, which we should investigate. > HoS hangs when job is empty > --- > > Key: HIVE-13525 > URL: https://issues.apache.org/jira/browse/HIVE-13525 > Project: Hive > Issue Type: Bug >Reporter: Rui Li >Assignee: Rui Li > > Observed in local tests. This should be the cause of HIVE-13402. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13505) Skip running TestDummy where possibe during precommit builds
[ https://issues.apache.org/jira/browse/HIVE-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243268#comment-15243268 ] Siddharth Seth commented on HIVE-13505: --- I was looking at the test output logs again and the ptest documenation. I believe the PTestClient just posts to the webserver - which then takes care of running the rest of the test. I don't think the webserver is getting a new build. Looks like a restart will be required. I'm not sure why we end up building ptest2 each time though - if this is how the system works. > Skip running TestDummy where possibe during precommit builds > > > Key: HIVE-13505 > URL: https://issues.apache.org/jira/browse/HIVE-13505 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13505.01.patch > > > On the main Hive build - this does nothing. There are some tests named > TestDummy under qtests - I'm not sure they do anything useful though. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13511) Run clidriver tests from within the qtest dir for the precommit tests
[ https://issues.apache.org/jira/browse/HIVE-13511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13511: -- Attachment: HIVE-13511.02.patch Updated patch, which changes the module name in case the driver is a spark driver to make sure it moves into the qtest-spark module instead of the qtest module. Looking at the ptest docs and logs again, a restart is likely required. > Run clidriver tests from within the qtest dir for the precommit tests > - > > Key: HIVE-13511 > URL: https://issues.apache.org/jira/browse/HIVE-13511 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13511.01.patch, HIVE-13511.02.patch, > example_maven-test.txt, example_testExecution.txt > > > The tests are currently run from the itests directory - which means there's > additional overhead of having to at least check whether files have changed. > Will attach a sample output - this adds up to 40+ seconds per batch. Getting > rid of this should be a reasonable saving overall. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13501) Invoke failure hooks if query fails on exception
[ https://issues.apache.org/jira/browse/HIVE-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13501: --- Attachment: HIVE-13501.2.patch Rebased patch 1 to master latest. > Invoke failure hooks if query fails on exception > > > Key: HIVE-13501 > URL: https://issues.apache.org/jira/browse/HIVE-13501 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13501.1.patch, HIVE-13501.2.patch > > > When a query fails on some exception, failure hooks are not called currently. > It's better to invoke such hooks so that we know the query is failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13478) [Cleanup] Improve HookUtils performance
[
https://issues.apache.org/jira/browse/HIVE-13478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reuben Kuhnert updated HIVE-13478:
--
Attachment: HIVE-13478.03.patch
> [Cleanup] Improve HookUtils performance
> ---
>
> Key: HIVE-13478
> URL: https://issues.apache.org/jira/browse/HIVE-13478
> Project: Hive
> Issue Type: Improvement
>Reporter: Reuben Kuhnert
>Assignee: Reuben Kuhnert
>Priority: Minor
> Attachments: HIVE-13478.01.patch, HIVE-13478.02.patch,
> HIVE-13478.03.patch
>
>
> Minor cleanup. {{HookUtils.getHooks}} is called multiple times for every
> statement executed performing nearly identical work. Cache the results of the
> work to improve performance (LRU).
> Also introduce the {{@CacheableHook}} annotation which can be appended to
> hooks that don't need to be re-instantiated using expensive reflection (such
> as Sentry hooks that load configuration on initialization).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13352) Seems unnecessary for HBase tests to call QTestUtil.tearDown to close zookeeper and others.
[ https://issues.apache.org/jira/browse/HIVE-13352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243236#comment-15243236 ] Siddharth Seth commented on HIVE-13352: --- Zookeeper is restarted between each and every test run by TestCliDriver (and it's variants). I don't think this is a big performance penalty, but seems like unnecessary step. We can definitely look at clearing up the contents of ZK instead of restarting it. > Seems unnecessary for HBase tests to call QTestUtil.tearDown to close > zookeeper and others. > --- > > Key: HIVE-13352 > URL: https://issues.apache.org/jira/browse/HIVE-13352 > Project: Hive > Issue Type: Improvement > Components: Test >Affects Versions: 2.1.0 >Reporter: Aihua Xu > > HBase tests TestHBaseCliDriver.java right now call QTestUtil.tearDown to turn > off Zookeeper and others after each test. Seems we can reuse them for all the > tests while we just need to clear all the test data similar to TestCliDriver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13473) upgrade Apache Directory Server version
[
https://issues.apache.org/jira/browse/HIVE-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243161#comment-15243161
]
Ashutosh Chauhan commented on HIVE-13473:
-
{{TestLdapAtnProviderWithMiniDS}} seems to be related. We need to fix that to
get this in.
> upgrade Apache Directory Server version
> ---
>
> Key: HIVE-13473
> URL: https://issues.apache.org/jira/browse/HIVE-13473
> Project: Hive
> Issue Type: Improvement
>Reporter: Gabor Liptak
>Assignee: Gabor Liptak
>Priority: Minor
> Attachments: HIVE-13473.2.patch, HIVE-13473.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13473) upgrade Apache Directory Server version
[
https://issues.apache.org/jira/browse/HIVE-13473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243152#comment-15243152
]
Hive QA commented on HIVE-13473:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798382/HIVE-13473.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9951 tests executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.maven.surefire.junit4.JUnit4Provider.org.apache.hive.service.auth.TestLdapAtnProviderWithMiniDS
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7604/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7604/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7604/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798382 - PreCommit-HIVE-TRUNK-Build
> upgrade Apache Directory Server version
> ---
>
> Key: HIVE-13473
> URL: https://issues.apache.org/jira/browse/HIVE-13473
> Project: Hive
> Issue Type: Improvement
>Reporter: Gabor Liptak
>Assignee: Gabor Liptak
>Priority: Minor
> Attachments: HIVE-13473.2.patch, HIVE-13473.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13352) Seems unnecessary for HBase tests to call QTestUtil.tearDown to close zookeeper and others.
[ https://issues.apache.org/jira/browse/HIVE-13352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-13352: Description: HBase tests TestHBaseCliDriver.java right now call QTestUtil.tearDown to turn off Zookeeper and others after each test. Seems we can reuse them for all the tests while we just need to clear all the test data similar to TestCliDriver. (was: HBase tests right now call QTestUtil.tearDown to turn off Zookeeper and others after each test. Seems we can reuse them for all the tests while we just need to clear all the test data similar to TestCliDriver.) > Seems unnecessary for HBase tests to call QTestUtil.tearDown to close > zookeeper and others. > --- > > Key: HIVE-13352 > URL: https://issues.apache.org/jira/browse/HIVE-13352 > Project: Hive > Issue Type: Improvement > Components: Test >Affects Versions: 2.1.0 >Reporter: Aihua Xu > > HBase tests TestHBaseCliDriver.java right now call QTestUtil.tearDown to turn > off Zookeeper and others after each test. Seems we can reuse them for all the > tests while we just need to clear all the test data similar to TestCliDriver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13352) Seems unnecessary for HBase tests to call QTestUtil.tearDown to close zookeeper and others.
[ https://issues.apache.org/jira/browse/HIVE-13352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243085#comment-15243085 ] Ashutosh Chauhan commented on HIVE-13352: - sounds like a good idea. cc: [~sseth] > Seems unnecessary for HBase tests to call QTestUtil.tearDown to close > zookeeper and others. > --- > > Key: HIVE-13352 > URL: https://issues.apache.org/jira/browse/HIVE-13352 > Project: Hive > Issue Type: Improvement > Components: Test >Affects Versions: 2.1.0 >Reporter: Aihua Xu > > HBase tests right now call QTestUtil.tearDown to turn off Zookeeper and > others after each test. Seems we can reuse them for all the tests while we > just need to clear all the test data similar to TestCliDriver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12041) Add unit test for HIVE-9386
[
https://issues.apache.org/jira/browse/HIVE-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243013#comment-15243013
]
Hive QA commented on HIVE-12041:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12765239/HIVE-12041.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 9968 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_in_file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_leftsemijoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union_with_udf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_after_multiple_inserts
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_update_all_types
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_mapjoin1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_div0
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_part_varchar
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_in_file
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7602/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7602/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7602/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12765239 - PreCommit-HIVE-TRUNK-Build
> Add unit test for HIVE-9386
> ---
>
> Key: HIVE-12041
> URL: https://issues.apache.org/jira/browse/HIVE-12041
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.0, 1.1.0, 1.1.1, 1.2.1
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12041.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6476) Support Append with Dynamic Partitioning
[ https://issues.apache.org/jira/browse/HIVE-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15243005#comment-15243005 ] Mariappan Asokan commented on HIVE-6476: I have the same question: What are the corner cases that need to be tested? Dynamic partitioning with append is a very common use case. Sushanth, if you can elaborate on the "corner cases" and give some pointers I can pick up this Jira and work on it. Thanks. > Support Append with Dynamic Partitioning > > > Key: HIVE-6476 > URL: https://issues.apache.org/jira/browse/HIVE-6476 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore, Query Processor, Thrift API >Reporter: Sushanth Sowmyan > > Currently, we do not support mixing dynamic partitioning and append in the > same job. One reason is that we need exhaustive testing of corner cases for > that, and a second reason is the behaviour of add_partitions. To support > dynamic partitioning with append, we'd have to have a > add_partitions_if_not_exist call, rather than an add_partitions call. > Thus, the current implementation in HIVE-6475 assumes immutability for all > dynamic partitioning jobs, irrespective of whether or not the table is marked > as mutable or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-13149) Remove some unnecessary HMS connections from HS2
[
https://issues.apache.org/jira/browse/HIVE-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu reopened HIVE-13149:
-
Seems it caused TestJdbcWithMiniHS2 to fail. Reverted the patch and will look
into that.
> Remove some unnecessary HMS connections from HS2
> -
>
> Key: HIVE-13149
> URL: https://issues.apache.org/jira/browse/HIVE-13149
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.1.0
>
> Attachments: HIVE-13149.1.patch, HIVE-13149.2.patch,
> HIVE-13149.3.patch, HIVE-13149.4.patch, HIVE-13149.5.patch,
> HIVE-13149.6.patch, HIVE-13149.7.patch
>
>
> In SessionState class, currently we will always try to get a HMS connection
> in {{start(SessionState startSs, boolean isAsync, LogHelper console)}}
> regardless of if the connection will be used later or not.
> When SessionState is accessed by the tasks in TaskRunner.java, although most
> of the tasks other than some like StatsTask, don't need to access HMS.
> Currently a new HMS connection will be established for each Task thread. If
> HiveServer2 is configured to run in parallel and the query involves many
> tasks, then the connections are created but unused.
> {noformat}
> @Override
> public void run() {
> runner = Thread.currentThread();
> try {
> OperationLog.setCurrentOperationLog(operationLog);
> SessionState.start(ss);
> runSequential();
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13499) TestJdbcWithMiniHS2 is hanging
[ https://issues.apache.org/jira/browse/HIVE-13499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242972#comment-15242972 ] Aihua Xu commented on HIVE-13499: - Yeah. That makes sense. My fault that I didn't pay attention to the test which didn't have an output. I will rework on the patch. I just reverted. Could you please remove the ignore flag? > TestJdbcWithMiniHS2 is hanging > -- > > Key: HIVE-13499 > URL: https://issues.apache.org/jira/browse/HIVE-13499 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Thejas M Nair >Assignee: Aihua Xu > Attachments: HIVE-13499.temp.patch, HIVE-13499.temp2.patch > > > After HIVE-13149 went in , TestJdbcWithMiniHS2. -testConcurrentStatements- > has been hanging, causing delays in the unit test run. > Update : looks like it might be affecting more than one test in this class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12019) Create unit test for HIVE-10732
[
https://issues.apache.org/jira/browse/HIVE-12019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242890#comment-15242890
]
Hive QA commented on HIVE-12019:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12764808/HIVE-12019.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7601/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7601/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7601/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-7601/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
833a7d1..6a776f5 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 833a7d1 HIVE-13418 : HiveServer2 HTTP mode should support
X-Forwarded-Host header for authorization/audits (Thejas Nair, reviewed by
Vaibhav Gumashta)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
+ git reset --hard origin/master
HEAD is now at 6a776f5 HIVE-13475: Allow aggregate functions in over clause
(Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12764808 - PreCommit-HIVE-TRUNK-Build
> Create unit test for HIVE-10732
> ---
>
> Key: HIVE-12019
> URL: https://issues.apache.org/jira/browse/HIVE-12019
> Project: Hive
> Issue Type: Bug
> Components: Tests
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-12019.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242888#comment-15242888
]
Hive QA commented on HIVE-13293:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12797965/HIVE-13293.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 9964 tests executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby7_noskew_multi_single_reducer
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7597/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7597/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7597/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12797965 - PreCommit-HIVE-TRUNK-Build
> Query occurs performance degradation after enabling parallel order by for
> Hive on Spark
> ---
>
> Key: HIVE-13293
> URL: https://issues.apache.org/jira/browse/HIVE-13293
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 2.0.0
>Reporter: Lifeng Wang
>Assignee: Rui Li
> Attachments: HIVE-13293.1.patch
>
>
> I use TPCx-BB to do some performance test on Hive on Spark engine. And found
> query 10 has performance degradation when enabling parallel order by.
> It seems that sampling cost much time before running the real query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13475) Allow aggregate functions in over clause
[
https://issues.apache.org/jira/browse/HIVE-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13475:
---
Resolution: Fixed
Fix Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks for the review [~ashutoshc]!
> Allow aggregate functions in over clause
>
>
> Key: HIVE-13475
> URL: https://issues.apache.org/jira/browse/HIVE-13475
> Project: Hive
> Issue Type: New Feature
> Components: Parser
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 2.1.0
>
> Attachments: HIVE-13475.patch
>
>
> Support to reference aggregate functions within the over clause needs to be
> added. For instance, currently the following query will fail:
> {noformat}
> select rank() over (order by sum(ws.c_int)) as return_rank
> from cbo_t3 ws
> group by ws.key;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11527) bypass HiveServer2 thrift interface for query results
[
https://issues.apache.org/jira/browse/HIVE-11527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242862#comment-15242862
]
Takanobu Asanuma commented on HIVE-11527:
-
Hi, [~sershe], [~vgumashta], and other experts.
I uploaded a new patch in Review Board just now. I think I have almost finished
implementing the features. So I'd like to explain the summary of all my
implementation.
*How to use the bypass*
When {{hive.server2.webhdfs.bypass.enabled}} is true, users can use the bypass.
The default is false.
*Tests*
I added some unit tests in {{TestJdbcWithMiniHS2}}, {{TestJdbcWithMiniMr}} and
{{TestJdbcWithMiniHA}}. They will help debugging.
*Changing thrift API*
I added three optional variables as the response from HS2 to JDBC drivers after
executing a query.
* {{finalDirUri}}: a pass of the directory which has the final data
* {{haConf}}: configurations for Namenode HA
* {{typeName}}: a type name for complex columns
*Decoding data*
Decoding data in clients side is implemented in {{HiveQueryResultSet}}. In the
latest patch, to avoid complex codes, clients can use the bypass only when the
final data is SequenceFile which is the default format of final data. I think
it is rare that clients change the default format.
*Handling HA*
When Namenode is HA, clients need some configurations which are in the cluster
side. They are passed in {{Driver#getFinalDirName}}.
*Unable to use the bypass*
In some cases, it is difficult to use the bypass. I wrote the cases in
{{TestJdbcWithMiniHS2#testUnableUseBypassCase}}. {{Driver#useBypass}} judges
whether clients use the bypass.
Some optimizations and bugs may remain. Please review the patch when you are
free.
Thank you very much for reading this long comment!
> bypass HiveServer2 thrift interface for query results
> -
>
> Key: HIVE-11527
> URL: https://issues.apache.org/jira/browse/HIVE-11527
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Sergey Shelukhin
>Assignee: Takanobu Asanuma
> Attachments: HIVE-11527.WIP.patch
>
>
> Right now, HS2 reads query results and returns them to the caller via its
> thrift API.
> There should be an option for HS2 to return some pointer to results (an HDFS
> link?) and for the user to read the results directly off HDFS inside the
> cluster, or via something like WebHDFS outside the cluster
> Review board link: https://reviews.apache.org/r/40867
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13415) Decouple Sessions from thrift binary transport
[
https://issues.apache.org/jira/browse/HIVE-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-13415:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed. Thanks [~prongs]
> Decouple Sessions from thrift binary transport
> --
>
> Key: HIVE-13415
> URL: https://issues.apache.org/jira/browse/HIVE-13415
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13415.01.patch, HIVE-13415.04.patch,
> HIVE-13415.05.patch
>
>
> Current behaviour is:
> * Open a thrift binary transport
> * create a session
> * close the transport
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13415) Decouple Sessions from thrift binary transport
[
https://issues.apache.org/jira/browse/HIVE-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu updated HIVE-13415:
---
Affects Version/s: (was: 2.0.0)
1.2.0
Fix Version/s: 2.1.0
Component/s: HiveServer2
> Decouple Sessions from thrift binary transport
> --
>
> Key: HIVE-13415
> URL: https://issues.apache.org/jira/browse/HIVE-13415
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.2.0
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Fix For: 2.1.0
>
> Attachments: HIVE-13415.01.patch, HIVE-13415.04.patch,
> HIVE-13415.05.patch
>
>
> Current behaviour is:
> * Open a thrift binary transport
> * create a session
> * close the transport
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13415) Decouple Sessions from thrift binary transport
[
https://issues.apache.org/jira/browse/HIVE-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242845#comment-15242845
]
Rajat Khandelwal commented on HIVE-13415:
-
After running the tests locally,
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager passes completely, so does
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs. Running the
MiniTezCliDriver tests with overwrite output=true doesn't change any files.
> Decouple Sessions from thrift binary transport
> --
>
> Key: HIVE-13415
> URL: https://issues.apache.org/jira/browse/HIVE-13415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13415.01.patch, HIVE-13415.04.patch,
> HIVE-13415.05.patch
>
>
> Current behaviour is:
> * Open a thrift binary transport
> * create a session
> * close the transport
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13287) Add logic to estimate stats for IN operator
[ https://issues.apache.org/jira/browse/HIVE-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13287: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Regenerated q files and pushed to master. Thanks for the review [~ashutoshc]! > Add logic to estimate stats for IN operator > --- > > Key: HIVE-13287 > URL: https://issues.apache.org/jira/browse/HIVE-13287 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13287.01.patch, HIVE-13287.02.patch, > HIVE-13287.03.patch, HIVE-13287.patch > > > Currently, IN operator is considered in the default case: reduces the input > rows to the half. This may lead to wrong estimates for the number of rows > produced by Filter operators. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13415) Decouple Sessions from thrift binary transport
[
https://issues.apache.org/jira/browse/HIVE-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242792#comment-15242792
]
Hive QA commented on HIVE-13415:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798463/HIVE-13415.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 9967 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_char_simple
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.concurrencyFalse
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLExclusive
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testRollback
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testWriteDynamicPartition
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7596/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7596/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7596/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798463 - PreCommit-HIVE-TRUNK-Build
> Decouple Sessions from thrift binary transport
> --
>
> Key: HIVE-13415
> URL: https://issues.apache.org/jira/browse/HIVE-13415
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Rajat Khandelwal
>Assignee: Rajat Khandelwal
> Attachments: HIVE-13415.01.patch, HIVE-13415.04.patch,
> HIVE-13415.05.patch
>
>
> Current behaviour is:
> * Open a thrift binary transport
> * create a session
> * close the transport
> Then the session gets closed. Consequently, all the operations running in the
> session also get killed.
> Whereas, if you open an HTTP transport, and close, the enclosing sessions are
> not closed.
> This seems like a bad design, having transport and sessions tightly coupled.
> I'd like to fix this.
> The issue that introduced it is
> [HIVE-9601|https://github.com/apache/hive/commit/48bea00c48853459af64b4ca9bfdc3e821c4ed82]
> Relevant discussions at
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223546=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223546],
>
> [here|https://issues.apache.org/jira/browse/HIVE-11485?focusedCommentId=15223827=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15223827]
> and mentioned links on those comments.
> Another thing that seems like a slightly bad design is this line of code in
> ThriftBinaryCLIService:
> {noformat}
> server.setServerEventHandler(serverEventHandler);
> {noformat}
> Whereas serverEventHandler is defined by the base class, with no users except
> one sub-class(ThriftBinaryCLIService), violating the separation of concerns.
--
This message was sent by Atlassian JIRA
[jira] [Resolved] (HIVE-13010) partitions autogenerated predicates broken
[ https://issues.apache.org/jira/browse/HIVE-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanilovsky Evgeny resolved HIVE-13010. --- Resolution: Not A Bug > partitions autogenerated predicates broken > -- > > Key: HIVE-13010 > URL: https://issues.apache.org/jira/browse/HIVE-13010 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Stanilovsky Evgeny >Priority: Trivial > > hi, i`m looking for simalar problem but found only: > https://issues.apache.org/jira/browse/HIVE-9630 > it`s looks like the same but you can easily repeat it on testing i hope. > I have two simalar requests , the difference in autogenerated data > predicates, in first case explain show ful lscan. > ''' > set hive.optimize.constant.propagation=true; > explain select * from logs.weather_forecasts where dt between > from_unixtime(unix_timestamp() - 3600*24*3, '-MM-dd') and > from_unixtime(unix_timestamp() - 3600*24*1, '-MM-dd') and > provider_id = 100 > STAGE PLANS: > 5 Stage: Stage-1 > 6 Map Reduce > 7 Map Operator Tree: > 8 TableScan > 9 alias: weather_forecasts > 10Statistics: Num rows: 36124837607 Data size: 47395787122046 > Basic stats: PARTIAL Column stats: NONE > > and > > set hive.optimize.constant.propagation=true; > explain select * from logs.redir_log where dt between > '2016-02-02' and > '2016-02-04' and > pid = 100 > 0 STAGE DEPENDENCIES: > 1 Stage-1 is a root stage > 2 Stage-0 depends on stages: Stage-1 > 3 > 4 STAGE PLANS: > 5 Stage: Stage-1 > 6 Map Reduce > 7 Map Operator Tree: > 8 TableScan > 9 alias: redir_log > 10Statistics: Num rows: 2798358420 Data size: 5761819991150 > Basic stats: COMPLETE Column stats: NONE > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13010) partitions autogenerated predicates broken
[ https://issues.apache.org/jira/browse/HIVE-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242790#comment-15242790 ] Stanilovsky Evgeny commented on HIVE-13010: --- ok, thanks a lot for your answers ! > partitions autogenerated predicates broken > -- > > Key: HIVE-13010 > URL: https://issues.apache.org/jira/browse/HIVE-13010 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Stanilovsky Evgeny >Priority: Trivial > > hi, i`m looking for simalar problem but found only: > https://issues.apache.org/jira/browse/HIVE-9630 > it`s looks like the same but you can easily repeat it on testing i hope. > I have two simalar requests , the difference in autogenerated data > predicates, in first case explain show ful lscan. > ''' > set hive.optimize.constant.propagation=true; > explain select * from logs.weather_forecasts where dt between > from_unixtime(unix_timestamp() - 3600*24*3, '-MM-dd') and > from_unixtime(unix_timestamp() - 3600*24*1, '-MM-dd') and > provider_id = 100 > STAGE PLANS: > 5 Stage: Stage-1 > 6 Map Reduce > 7 Map Operator Tree: > 8 TableScan > 9 alias: weather_forecasts > 10Statistics: Num rows: 36124837607 Data size: 47395787122046 > Basic stats: PARTIAL Column stats: NONE > > and > > set hive.optimize.constant.propagation=true; > explain select * from logs.redir_log where dt between > '2016-02-02' and > '2016-02-04' and > pid = 100 > 0 STAGE DEPENDENCIES: > 1 Stage-1 is a root stage > 2 Stage-0 depends on stages: Stage-1 > 3 > 4 STAGE PLANS: > 5 Stage: Stage-1 > 6 Map Reduce > 7 Map Operator Tree: > 8 TableScan > 9 alias: redir_log > 10Statistics: Num rows: 2798358420 Data size: 5761819991150 > Basic stats: COMPLETE Column stats: NONE > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[
https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242738#comment-15242738
]
Jesus Camacho Rodriguez commented on HIVE-13518:
LGTM, +1.
Nitpick: {{getMapJoinConversionPos}} could be renamed, as it does not apply
only to MapJoin anymore (getBigTablePos?)
> Hive on Tez: Shuffle joins do not choose the right 'big' table.
> ---
>
> Key: HIVE-13518
> URL: https://issues.apache.org/jira/browse/HIVE-13518
> Project: Hive
> Issue Type: Bug
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13518.1.patch
>
>
> Currently the big table is always assumed to be at position 0 but this isn't
> efficient for some queries as the big table at position 1 could have a lot
> more keys/skew. We already have a mechanism of choosing the big table that
> can be leveraged to make the right choice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242622#comment-15242622
]
Hive QA commented on HIVE-13290:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798915/HIVE-13290.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 4 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/142/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/142/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-METASTORE-Test-142/
Messages:
{noformat}
LXC derby found.
LXC derby is not started. Starting container...
Container started.
Preparing derby container...
Container prepared.
Calling /hive/testutils/metastore/dbs/derby/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/derby/execute.sh ...
Tests executed.
LXC mysql found.
LXC mysql is not started. Starting container...
Container started.
Preparing mysql container...
Container prepared.
Calling /hive/testutils/metastore/dbs/mysql/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/mysql/execute.sh ...
Tests executed.
LXC oracle found.
LXC oracle is not started. Starting container...
Container started.
Preparing oracle container...
Container prepared.
Calling /hive/testutils/metastore/dbs/oracle/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/oracle/execute.sh ...
Tests executed.
LXC postgres found.
LXC postgres is not started. Starting container...
Container started.
Preparing postgres container...
Container prepared.
Calling /hive/testutils/metastore/dbs/postgres/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/postgres/execute.sh ...
Tests executed.
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798915 - PreCommit-HIVE-METASTORE-Test
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id),
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id)
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id)
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13349) Metastore Changes : API calls for retrieving primary keys and foreign keys information
[
https://issues.apache.org/jira/browse/HIVE-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242616#comment-15242616
]
Hive QA commented on HIVE-13349:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798914/HIVE-13349.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 4 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/141/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-METASTORE-Test/141/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-METASTORE-Test-141/
Messages:
{noformat}
LXC derby found.
LXC derby is not started. Starting container...
Container started.
Preparing derby container...
Container prepared.
Calling /hive/testutils/metastore/dbs/derby/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/derby/execute.sh ...
Tests executed.
LXC mysql found.
LXC mysql is not started. Starting container...
Container started.
Preparing mysql container...
Container prepared.
Calling /hive/testutils/metastore/dbs/mysql/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/mysql/execute.sh ...
Tests executed.
LXC oracle found.
LXC oracle is not started. Starting container...
Container started.
Preparing oracle container...
Container prepared.
Calling /hive/testutils/metastore/dbs/oracle/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/oracle/execute.sh ...
Tests executed.
LXC postgres found.
LXC postgres is not started. Starting container...
Container started.
Preparing postgres container...
Container prepared.
Calling /hive/testutils/metastore/dbs/postgres/prepare.sh ...
Server prepared.
Calling /hive/testutils/metastore/dbs/postgres/execute.sh ...
Tests executed.
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798914 - PreCommit-HIVE-METASTORE-Test
> Metastore Changes : API calls for retrieving primary keys and foreign keys
> information
> --
>
> Key: HIVE-13349
> URL: https://issues.apache.org/jira/browse/HIVE-13349
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: 13449.2.patch, HIVE-13349.1.patch, HIVE-13349.3.patch,
> HIVE-13349.4.patch, HIVE-13349.5.patch, HIVE-13349.6.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13084) Vectorization add support for PROJECTION Multi-AND/OR
[
https://issues.apache.org/jira/browse/HIVE-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242611#comment-15242611
]
Hive QA commented on HIVE-13084:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798860/HIVE-13084.06.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9968 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_multi_and_projection
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7595/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7595/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7595/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798860 - PreCommit-HIVE-TRUNK-Build
> Vectorization add support for PROJECTION Multi-AND/OR
> -
>
> Key: HIVE-13084
> URL: https://issues.apache.org/jira/browse/HIVE-13084
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Rajesh Balamohan
>Assignee: Matt McCline
> Attachments: HIVE-13084.01.patch, HIVE-13084.02.patch,
> HIVE-13084.03.patch, HIVE-13084.04.patch, HIVE-13084.05.patch,
> HIVE-13084.06.patch, vector_between_date.q
>
>
> When there is case statement in group by, hive throws unable to vectorize
> exception.
> e.g query just to demonstrate the problem
> {noformat}
> explain select l_partkey, case when l_commitdate between '2015-06-30' AND
> '2015-07-06' THEN '2015-06-30' END as wk from lineitem_test_l_shipdate_ts
> group by l_partkey, case when l_commitdate between '2015-06-30' AND
> '2015-07-06' THEN '2015-06-30' END;
> org.apache.hadoop.hive.ql.metadata.HiveException: Could not vectorize
> expression: org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc
> Vertex dependency in root stage
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> Stage-0
> Fetch Operator
> limit:-1
> Stage-1
> Reducer 2
> File Output Operator [FS_7]
> Group By Operator [GBY_5] (rows=888777234 width=108)
> Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
> <-Map 1 [SIMPLE_EDGE]
> SHUFFLE [RS_4]
> PartitionCols:_col0, _col1
> Group By Operator [GBY_3] (rows=1777554469 width=108)
> Output:["_col0","_col1"],keys:_col0, _col1
> Select Operator [SEL_1] (rows=1777554469 width=108)
> Output:["_col0","_col1"]
> TableScan [TS_0] (rows=1777554469 width=108)
>
> rajesh@lineitem_test_l_shipdate_ts,lineitem_test_l_shipdate_ts,Tbl:COMPLETE,Col:NONE,Output:["l_partkey","l_commitdate"]
> {noformat}
> \cc [~mmccline], [~gopalv]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13290:
-
Status: Open (was: Patch Available)
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id),
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id)
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id)
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13290:
-
Status: Patch Available (was: Open)
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id),
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id)
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id)
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13290) Support primary keys/foreign keys constraint as part of create table command in Hive
[
https://issues.apache.org/jira/browse/HIVE-13290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan updated HIVE-13290:
-
Attachment: HIVE-13290.5.patch
> Support primary keys/foreign keys constraint as part of create table command
> in Hive
>
>
> Key: HIVE-13290
> URL: https://issues.apache.org/jira/browse/HIVE-13290
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13290.1.patch, HIVE-13290.2.patch,
> HIVE-13290.3.patch, HIVE-13290.4.patch, HIVE-13290.5.patch
>
>
> SUPPORT for the following statements
> {code}
> CREATE TABLE product
> (
> product_idINTEGER,
> product_vendor_id INTEGER,
> PRIMARY KEY (product_id),
> CONSTRAINT product_fk_1 FOREIGN KEY (product_vendor_id) REFERENCES
> vendor(vendor_id)
> );
> CREATE TABLE vendor
> (
> vendor_id INTEGER,
> PRIMARY KEY (vendor_id)
> );
> {code}
> In the above syntax, [CONSTRAINT constraint-Name] is optional. If this is not
> specified by the user, we will use system generated constraint name. For the
> purpose of simplicity, we will allow CONSTRAINT option for foreign keys and
> not primary key since there is only one primary key per table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13349) Metastore Changes : API calls for retrieving primary keys and foreign keys information
[ https://issues.apache.org/jira/browse/HIVE-13349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13349: - Attachment: HIVE-13349.6.patch > Metastore Changes : API calls for retrieving primary keys and foreign keys > information > -- > > Key: HIVE-13349 > URL: https://issues.apache.org/jira/browse/HIVE-13349 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: 13449.2.patch, HIVE-13349.1.patch, HIVE-13349.3.patch, > HIVE-13349.4.patch, HIVE-13349.5.patch, HIVE-13349.6.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13499) TestJdbcWithMiniHS2 is hanging
[ https://issues.apache.org/jira/browse/HIVE-13499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242579#comment-15242579 ] Thejas M Nair commented on HIVE-13499: -- Yes, I will take a look at those patches. However, leaving these tests disabled would not be a good idea, we might have more regressions getting committed that we will not notice for long time. For example, HIVE-12049 is adding new tests in same class. I think we should revert the optimization inn HIVE-13149 temporarily because of the test case breakage and possible other production issues it might bring about (as we haven't really nailed down the interaction). We can bring it back after the changes you are making to make things thread safe (or other fixes to get the test working). Does that sound OK ? > TestJdbcWithMiniHS2 is hanging > -- > > Key: HIVE-13499 > URL: https://issues.apache.org/jira/browse/HIVE-13499 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Thejas M Nair >Assignee: Aihua Xu > Attachments: HIVE-13499.temp.patch, HIVE-13499.temp2.patch > > > After HIVE-13149 went in , TestJdbcWithMiniHS2. -testConcurrentStatements- > has been hanging, causing delays in the unit test run. > Update : looks like it might be affecting more than one test in this class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13490) Change itests to be part of the main Hive build
[ https://issues.apache.org/jira/browse/HIVE-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242574#comment-15242574 ] Siddharth Seth commented on HIVE-13490: --- Thanks for the pointer [~ashutoshc]. From the thread, I'm still a little unclear on why we don't want to link the two projects. Is it to avoid a long test run if someone accidentally runs a mvn test ? Don't really see too much more interest here, but I'd imagine this is useful for Hive developers (at least new ones). I'm guessing everyone has their existing workflows to modify itests which work for them. > Change itests to be part of the main Hive build > --- > > Key: HIVE-13490 > URL: https://issues.apache.org/jira/browse/HIVE-13490 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13490.01.patch > > > Instead of having to build Hive, and then itests separately. > With IntelliJ, this ends up being loaded as two separate dependencies, and > there's a lot of hops involved to make changes. > Does anyone know why these have been kept separate ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13511) Run clidriver tests from within the qtest dir for the precommit tests
[ https://issues.apache.org/jira/browse/HIVE-13511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242561#comment-15242561 ] Siddharth Seth commented on HIVE-13511: --- Same comment as HIVE-13505, lets commit and monitor a run. This one would need a dedicated run to itself. If it's high enough in the build queue - this can wait till it's turn. > Run clidriver tests from within the qtest dir for the precommit tests > - > > Key: HIVE-13511 > URL: https://issues.apache.org/jira/browse/HIVE-13511 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13511.01.patch, example_maven-test.txt, > example_testExecution.txt > > > The tests are currently run from the itests directory - which means there's > additional overhead of having to at least check whether files have changed. > Will attach a sample output - this adds up to 40+ seconds per batch. Getting > rid of this should be a reasonable saving overall. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13520) Don't allow any test to run for longer than 45minutes in the ptest setup
[ https://issues.apache.org/jira/browse/HIVE-13520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242560#comment-15242560 ] Siddharth Seth commented on HIVE-13520: --- HIVE-13507 adds some logging which should help figure out a better value. If not 45 minutes, do you think 1 hour is reasonable ? 15 tests running for 45 minutes seems like a lot - especially when entire runs can complete in less than 2 hours (<1.5 hour test runtime). This is mainly to prevent a bad test from causing the entire test-run to take 3+ hours. HostExecutor.java actually writes out a file based on patch-exec.vm - so I won't be surprised if this works without a restart. As with HIVE-13505, lets get this committed, and monitor a couple of runs to see what happens. > Don't allow any test to run for longer than 45minutes in the ptest setup > > > Key: HIVE-13520 > URL: https://issues.apache.org/jira/browse/HIVE-13520 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13520.01.txt, HIVE-13520.02.txt > > > Current timeout for batches is 2hours. This needs to be lowered. 1hour may be > too much as well. We can start with this, and reduce timeouts further. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13505) Skip running TestDummy where possibe during precommit builds
[
https://issues.apache.org/jira/browse/HIVE-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242548#comment-15242548
]
Siddharth Seth commented on HIVE-13505:
---
Missed the most relevant line in the output, which would indicate that this
file is generated each time
{code}
2016-04-12 09:39:04,069 INFO TestCheckPhase.execute:58 Searching line : +++
b/.travis.yml
2016-04-12 09:39:04,070 INFO PTest.run:169 Executing
org.apache.hive.ptest.execution.PrepPhase
2016-04-12 09:39:04,070 INFO LocalCommand.:35 Starting rm -rf
/data/hive-ptest/working/scratch
2016-04-12 09:39:04,075 INFO LocalCommand.:35 Starting mkdir -p
/data/hive-ptest/working/scratch
2016-04-12 09:39:04,142 INFO LocalCommand.:35 Starting cp -f
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-7556/patches/PreCommit-HIVE-TRUNK-Build-7556.patch
/data/hive-ptest/working/scratch/build.patch
2016-04-12 09:39:04,180 INFO LocalCommand.:35 Starting bash
/data/hive-ptest/working/scratch/source-prep.sh
{code}
> Skip running TestDummy where possibe during precommit builds
>
>
> Key: HIVE-13505
> URL: https://issues.apache.org/jira/browse/HIVE-13505
> Project: Hive
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Fix For: 2.1.0
>
> Attachments: HIVE-13505.01.patch
>
>
> On the main Hive build - this does nothing. There are some tests named
> TestDummy under qtests - I'm not sure they do anything useful though.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13522) regexp_extract.q hangs on master
[ https://issues.apache.org/jira/browse/HIVE-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242546#comment-15242546 ] Siddharth Seth commented on HIVE-13522: --- Marking as blocker to track re-enabling the test for 2.1 > regexp_extract.q hangs on master > > > Key: HIVE-13522 > URL: https://issues.apache.org/jira/browse/HIVE-13522 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Blocker > Attachments: HIVE-13522.patch, jstack_regexp_extract.txt > > > Disable to unblock Hive QA runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13522) regexp_extract.q hangs on master
[ https://issues.apache.org/jira/browse/HIVE-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13522: -- Target Version/s: 2.1.0 Priority: Blocker (was: Major) > regexp_extract.q hangs on master > > > Key: HIVE-13522 > URL: https://issues.apache.org/jira/browse/HIVE-13522 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan >Priority: Blocker > Attachments: HIVE-13522.patch, jstack_regexp_extract.txt > > > Disable to unblock Hive QA runs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13505) Skip running TestDummy where possibe during precommit builds
[ https://issues.apache.org/jira/browse/HIVE-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13505: -- Status: Patch Available (was: Reopened) > Skip running TestDummy where possibe during precommit builds > > > Key: HIVE-13505 > URL: https://issues.apache.org/jira/browse/HIVE-13505 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13505.01.patch > > > On the main Hive build - this does nothing. There are some tests named > TestDummy under qtests - I'm not sure they do anything useful though. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13507) Improved logging for ptest
[ https://issues.apache.org/jira/browse/HIVE-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13507: -- Attachment: HIVE-13507.01.patch Simple set of changes to log the amount of time each batch takes (can be inferred today by a diff between the rsync and execution of a batch), the time at which a host finished (figure out outliers), number of batches processed etc. cc [~ashutoshc], [~szehon] for review. I don't think this needs a build system restart since the ptest2 framework is built each time. I could be wrong though. > Improved logging for ptest > -- > > Key: HIVE-13507 > URL: https://issues.apache.org/jira/browse/HIVE-13507 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13507.01.patch > > > Include information about batch runtimes, outlier lists, host completion > times, etc. Try identifying tests which cause the build to take a long time > while holding onto resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13507) Improved logging for ptest
[ https://issues.apache.org/jira/browse/HIVE-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13507: -- Status: Patch Available (was: Open) > Improved logging for ptest > -- > > Key: HIVE-13507 > URL: https://issues.apache.org/jira/browse/HIVE-13507 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13507.01.patch > > > Include information about batch runtimes, outlier lists, host completion > times, etc. Try identifying tests which cause the build to take a long time > while holding onto resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13507) Improved logging for ptest
[ https://issues.apache.org/jira/browse/HIVE-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-13507: -- Assignee: Siddharth Seth Target Version/s: 2.1.0 > Improved logging for ptest > -- > > Key: HIVE-13507 > URL: https://issues.apache.org/jira/browse/HIVE-13507 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > > Include information about batch runtimes, outlier lists, host completion > times, etc. Try identifying tests which cause the build to take a long time > while holding onto resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13287) Add logic to estimate stats for IN operator
[
https://issues.apache.org/jira/browse/HIVE-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242504#comment-15242504
]
Hive QA commented on HIVE-13287:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798015/HIVE-13287.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 9964 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_non_ascii_literal2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_grouping_sets
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_interval_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join_filters
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query17
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query29
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query46
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query89
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_single_reducer3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7594/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7594/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7594/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798015 - PreCommit-HIVE-TRUNK-Build
> Add logic to estimate stats for IN operator
> ---
>
> Key: HIVE-13287
> URL: https://issues.apache.org/jira/browse/HIVE-13287
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13287.01.patch, HIVE-13287.02.patch,
> HIVE-13287.03.patch, HIVE-13287.patch
>
>
> Currently, IN operator is considered in the default case: reduces the input
> rows to the half. This may lead to wrong estimates for the number of rows
> produced by Filter operators.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
