Re: Time to Remove Hive-on-Spark

[Gopal V]

+1

Cheers,
Gopal

On 6/3/20 7:48 PM, Jesus Camacho Rodriguez wrote:

+1

-Jesús

On Wed, Jun 3, 2020 at 1:58 PM Alan Gates  wrote:


+1.

Alan.

On Wed, Jun 3, 2020 at 1:40 PM Prasanth Jayachandran
 wrote:


+1


On Jun 3, 2020, at 1:38 PM, Ashutosh Chauhan 

wrote:


+1

On Wed, Jun 3, 2020 at 1:23 PM David Mollitor 

wrote:



Hello Gang,

I have spent some time working on upgrading Avro (far less than

others):


https://issues.apache.org/jira/browse/HIVE-21737

This should be a relatively easy thing to do, but is blocked by
Hive-on-Spark.  HoS has a weird thing where it downloads some
cloud-storage-hosted file of Spark-Hadoop as part of its maven run.

Since HoS is not going to receive updates from the major vendors, is

it

time to simply remove it?

Tests are currently disabled:
https://issues.apache.org/jira/browse/HIVE-23137

Thanks.

Re: [VOTE] Should we release Hive Storage API 2.7.2-rc1?

[Gopal V]

Hi,

Validated checksums, signatures, built and verified against latest orc.

Since I was lazy enough to automate this, here's a script for others who 
might not have voted (or want to add things to this).


https://github.com/t3rmin4t0r/verify-asf-releases + make -f 
Makefile.storage-api


should do all the checks I did, unattended.

Cheers,
Gopal

On 5/5/20 7:49 AM, Jesus Camacho Rodriguez wrote:

All,

I'd like to make a storage-api release with HIVE-22959
 and HIVE-23215 <
https://issues.apache.org/jira/browse/HIVE-23215> in it.

Should we release the following artifacts as Hive Storage API 2.7.2?

tar: http://home.apache.org/~jcamacho/hive-storage-2.7.2/
tag: https://github.com/apache/hive/releases/tag/storage-release-2.7.2-rc1
jiras: https://issues.apache.org/jira/projects/HIVE/versions/12347828

Thanks!

-Jesús

Re: Review Request 71995: TopN Key optimizer should use array instead of priority queue

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71995/#review219348
---




ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyFilter.java
Line 42 (original), 44 (patched)
<https://reviews.apache.org/r/71995/#comment307492>

Add a counter for metrics of this


- Gopal V


On Jan. 14, 2020, 3:38 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71995/
> ---
> 
> (Updated Jan. 14, 2020, 3:38 p.m.)
> 
> 
> Review request for hive, Gopal V, Jesús Camacho Rodríguez, and Krisztian Kasa.
> 
> 
> Bugs: HIVE-22726
> https://issues.apache.org/jira/browse/HIVE-22726
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The TopN key optimizer currently uses a priority queue for keeping track of 
> the largest/smallest rows. Its max size is the same as the user specified 
> limit. This should be replaced a more cache line friendly array with a small 
> (128) maximum size and see how much performance is gained.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e7724f9084f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyFilter.java 4998766f064 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java b7c12502204 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorTopNKeyOperator.java 
> 5faa038c18d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java 
> ce6efa49192 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java ff815434f0c 
> 
> 
> Diff: https://reviews.apache.org/r/71995/diff/1/
> 
> 
> Testing
> ---
> 
> with the following query:
> 
> 
> use tpcds_bin_partitioned_orc_100;
> set hive.optimize.topnkey=true;
> set hive.optimize.topnkey.max=5;
> 
> select  i_item_id,
> s_state, grouping(s_state) g_state,
> avg(ss_quantity) agg1,
> avg(ss_list_price) agg2,
> avg(ss_coupon_amt) agg3,
> avg(ss_sales_price) agg4
>  from store_sales, customer_demographics, date_dim, store, item
>  where ss_sold_date_sk = d_date_sk and
>ss_item_sk = i_item_sk and
>ss_store_sk = s_store_sk and
>ss_cdemo_sk = cd_demo_sk
>  group by rollup (i_item_id, s_state)
>  order by i_item_id
>  ,s_state
>  limit 5;
> 
> 
> Results:
>   enabled:   5 rows selected (715.26 seconds)
>   enabled:   5 rows selected (605.888 seconds)
>   disabled:  5 rows selected (1208.168 seconds)
>   disabled:  5 rows selected (1219.482 seconds)
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

[jira] [Created] (HIVE-22169) Tez: SplitGenerator tries to look for plan files which won't exist for Tez

[Gopal V (Jira)]

Gopal V created HIVE-22169:
--

 Summary: Tez: SplitGenerator tries to look for plan files which 
won't exist for Tez
 Key: HIVE-22169
 URL: https://issues.apache.org/jira/browse/HIVE-22169
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


{code}
at 
org.apache.hadoop.hive.ql.exec.Utilities.clearWork(Utilities.java:310)
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:318)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method
{code}

The split generator tries to clear out the work items from HDFS, which will 
never exist for Tez plans.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (HIVE-22163) CBO: Enabling CBO turns on stats estimation, even when the estimation is disabled

[Gopal V (Jira)]

Gopal V created HIVE-22163:
--

 Summary: CBO: Enabling CBO turns on stats estimation, even when 
the estimation is disabled
 Key: HIVE-22163
 URL: https://issues.apache.org/jira/browse/HIVE-22163
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


{code}
create table claims(claim_rec_id bigint, claim_invoice_num string, typ_c int);
alter table claims update statistics set 
('numRows'='1154941534','rawDataSize'='1135307527922');


set hive.stats.estimate=false;

explain extended select count(1) from claims where typ_c=3;

set hive.stats.ndv.estimate.percent=5e-7;

explain extended select count(1) from claims where typ_c=3;
{code}

Expecting the standard /2 for the single filter, but we instead get 5 rows.

{code}
'Map Operator Tree:'
'TableScan'
'  alias: claims'
'  filterExpr: (typ_c = 3) (type: boolean)'
'  Statistics: Num rows: 1154941534 Data size: 4388777832 Basic 
stats: COMPLETE Column stats: NONE'
'  GatherStats: false'
'  Filter Operator'
'isSamplingPred: false'
'predicate: (typ_c = 3) (type: boolean)'
'Statistics: Num rows: 5 Data size: 19 Basic stats: 
COMPLETE Column stats: NONE'
{code}

The estimation is in effect, as changing the estimate.percent changes this.

{code}
'  filterExpr: (typ_c = 3) (type: boolean)'
'  Statistics: Num rows: 1154941534 Data size: 4388777832 Basic 
stats: COMPLETE Column stats: NONE'
'  GatherStats: false'
'  Filter Operator'
'isSamplingPred: false'
'predicate: (typ_c = 3) (type: boolean)'
'Statistics: Num rows: 230988307 Data size: 877755567 Basic 
stats: COMPLETE Column stats: NONE'
{code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (HIVE-22161) UDF: FunctionRegistry should cache statefulness of UDF

[Gopal V (Jira)]

Gopal V created HIVE-22161:
--

 Summary: UDF: FunctionRegistry should cache statefulness of UDF
 Key: HIVE-22161
 URL: https://issues.apache.org/jira/browse/HIVE-22161
 Project: Hive
  Issue Type: Bug
  Components: UDF
Reporter: Gopal V


There's a hidden synchronization across threads when looking up isStateful and 
isDeterministic.

https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27

{code}
  // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142
  public static  T getAnnotation(Class clazz, Class 
annotationClass) {
synchronized (annotationClass) {
  return clazz.getAnnotation(annotationClass);
}
  }
{code}

This is serializing multiple threads initializing UDFs (or checking them during 
compilation) & also being locked across threads for each instance of 
GenericUDFOpEqual in the specific scenario.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (HIVE-22106) PCR: Remove cross-query synchronization for the partition-eval

[Gopal V (JIRA)]

Gopal V created HIVE-22106:
--

 Summary: PCR: Remove cross-query synchronization for the 
partition-eval 
 Key: HIVE-22106
 URL: https://issues.apache.org/jira/browse/HIVE-22106
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


{code}
HiveServer2-Handler-Pool: Thread-492  Blocked CPU usage on sample: 0ms
  
org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.evalExprWithPart(ExprNodeDesc,
 Partition, List, StructObjectInspector) PartExprEvalUtils.java:58
  
org.apache.hadoop.hive.ql.optimizer.pcr.PcrExprProcFactory.evalExprWithPart(ExprNodeDesc,
 Partition, List) PcrExprProcFactory.java:83
  
org.apache.hadoop.hive.ql.optimizer.pcr.PcrExprProcFactory$GenericFuncExprProcessor.handleDeterministicUdf(PcrExprProcCtx,
 ExprNodeGenericFuncDesc, Object[]) PcrExprProcFactory.java:317
  
org.apache.hadoop.hive.ql.optimizer.pcr.PcrExprProcFactory$GenericFuncExprProcessor.process(Node,
 Stack, NodeProcessorCtx, Object[]) PcrExprProcFactory.java:298
  org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(Node, Stack, 
Object[]) DefaultRuleDispatcher.java:90
  org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(Node, 
Stack) DefaultGraphWalker.java:105
  org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(Node, Stack) 
DefaultGraphWalker.java:89
  org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(Node) 
DefaultGraphWalker.java:158
  org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(Collection, 
HashMap) DefaultGraphWalker.java:120
{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (HIVE-22076) JDK11: Remove ParallelGC in debug.sh

[Gopal V (JIRA)]

Gopal V created HIVE-22076:
--

 Summary: JDK11: Remove ParallelGC in debug.sh
 Key: HIVE-22076
 URL: https://issues.apache.org/jira/browse/HIVE-22076
 Project: Hive
  Issue Type: Bug
  Components: Diagnosability
Affects Versions: 4.0.0
Reporter: Gopal V


The JDK debug mode no longer depends on ParallelGC 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (HIVE-22064) CBO: Rewrite year() predicate to a constant condition

[Gopal V (JIRA)]

Gopal V created HIVE-22064:
--

 Summary: CBO: Rewrite year() predicate to a constant condition
 Key: HIVE-22064
 URL: https://issues.apache.org/jira/browse/HIVE-22064
 Project: Hive
  Issue Type: Improvement
Reporter: Gopal V


{code}
CREATE TEMPORARY TABLE users (signup_date date, user_id bigint) stored as ORC;

INSERT INTO users values("1999-01-01", 1);

EXPLAIN ANALYZE
SELECT year(signup_date), count(distinct user_id) from users where 
year(signup_date) BETWEEN 2017 and 2019
GROUP BY year(signup_date);
{code}

The YEAR() or EXTRACT( YEAR) is not rewritten into a constant for push-down.

{code}
EXPLAIN ANALYZE
SELECT year(signup_date), count(distinct user_id) from users where 
signup_date BETWEEN DATE'2017-01-01' and DATE'2019-12-31'
GROUP BY year(signup_date);
{code}

Does push-down into the storage layers.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (HIVE-21985) LLAP IO: Log schema evolution incompatibilities at WARN level always

[Gopal V (JIRA)]

Gopal V created HIVE-21985:
--

 Summary: LLAP IO: Log schema evolution incompatibilities at WARN 
level always
 Key: HIVE-21985
 URL: https://issues.apache.org/jira/browse/HIVE-21985
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


When reading incompatible schema, LLAP IO simply skips over the file and does 
not cache it.

The logging at WARN level would be useful and simplify the root-cause via logs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21828) Tez: Use a TezStaticConfiguration inherited from TezRuntimeConfiguration

[Gopal V (JIRA)]

Gopal V created HIVE-21828:
--

 Summary: Tez: Use a TezStaticConfiguration inherited from 
TezRuntimeConfiguration
 Key: HIVE-21828
 URL: https://issues.apache.org/jira/browse/HIVE-21828
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


The HS2 tez-site.xml does not change dynamically - the XML parsed components of 
the config can be obtained statically and kept across sessions.

This allows for the replacing of "new TezConfiguration()" with a HS2 local 
version instead.

The configuration object however has to reference the right resource file (i.e 
location of tez-site.xml) without reparsing it for each query.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21819) HiveConf: differentiate the resource between file configs and explicit sets

[Gopal V (JIRA)]

Gopal V created HIVE-21819:
--

 Summary: HiveConf: differentiate the resource between file configs 
and explicit sets
 Key: HIVE-21819
 URL: https://issues.apache.org/jira/browse/HIVE-21819
 Project: Hive
  Issue Type: Bug
  Components: Configuration
Affects Versions: 4.0.0
Reporter: Gopal V






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21818) CBO: Copying TableRelOptHiveTable has metastore traffic

[Gopal V (JIRA)]

Gopal V created HIVE-21818:
--

 Summary: CBO: Copying TableRelOptHiveTable has metastore traffic
 Key: HIVE-21818
 URL: https://issues.apache.org/jira/browse/HIVE-21818
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


While the optimizer is running, whenever it makes a copy of the TableScan to 
perform potential rewrites, there is Hive metastore traffic from inside CBO 
optimizer.

{code}
  public RelOptHiveTable(RelOptSchema calciteSchema, RelDataTypeFactory 
typeFactory, List qualifiedTblName,
  RelDataType rowType, Table hiveTblMetadata, List 
hiveNonPartitionCols,
  List hivePartitionCols, List hiveVirtualCols, 
HiveConf hconf,
  Map partitionCache, Map colStatsCache,
  AtomicInteger noColsMissingStats) { 


Pair, List> constraintKeys = 
generateKeys();
this.keys = constraintKeys.left;
this.nonNullablekeys = constraintKeys.right;
this.referentialConstraints = generateReferentialConstraints();
}
{code}

This is triggered from the rules as the partition pruner

{code}
  protected void perform(RelOptRuleCall call, Filter filter,
  HiveTableScan tScan) {
// Original table
RelOptHiveTable hiveTable = (RelOptHiveTable) tScan.getTable();

// Copy original table scan and table
HiveTableScan tScanCopy = tScan.copyIncludingTable(tScan.getRowType());
RelOptHiveTable hiveTableCopy = (RelOptHiveTable) tScanCopy.getTable();
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21817) CBO: PartitionPruner cache-miss from Calcite planner

[Gopal V (JIRA)]

Gopal V created HIVE-21817:
--

 Summary: CBO: PartitionPruner cache-miss from Calcite planner
 Key: HIVE-21817
 URL: https://issues.apache.org/jira/browse/HIVE-21817
 Project: Hive
  Issue Type: Bug
  Components: CBO
Reporter: Gopal V
 Attachments: calcite-ppr.png

 !calcite-ppr.png! 

{code}
public static PrunedPartitionList prune(Table tab, ExprNodeDesc prunerExpr,
  HiveConf conf, String alias, Map 
prunedPartitionsMap)
  throws SemanticException {

vs

  public PrunedPartitionList getPrunedPartitions(String alias, 
TableScanOperator ts)
  throws SemanticException {
PrunedPartitionList partsList = opToPartList.get(ts);
{code}

Only the 2nd one hits the PPR cache.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21805) HiveServer2: Use the fast ShutdownHookManager APIs

[Gopal V (JIRA)]

Gopal V created HIVE-21805:
--

 Summary: HiveServer2: Use the fast ShutdownHookManager APIs
 Key: HIVE-21805
 URL: https://issues.apache.org/jira/browse/HIVE-21805
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V
 Attachments: shutdownhookmanager-configuration.png

Hadoop ShutDownHookManager calls "new Configuration()" inside if the parameters 
are not provided as args.

This unzips jars & looks for the .xml files in the entire classpath.

 !shutdownhookmanager-configuration.png! 

Hive has its own impl of ShutDownHookManager, which has also history from the 
hadoo one (added to code instead of shims).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21800) Tez: Cartesian product reparses HiveConf XML

[Gopal V (JIRA)]

Gopal V created HIVE-21800:
--

 Summary: Tez: Cartesian product reparses HiveConf XML 
 Key: HIVE-21800
 URL: https://issues.apache.org/jira/browse/HIVE-21800
 Project: Hive
  Issue Type: Bug
  Components: Tez
Reporter: Gopal V
 Attachments: Tez-CartesianProductSlowness.png

{code}
  CartesianProductConfig cpConfig = new 
CartesianProductConfig(crossProductSources);
  edgeManagerDescriptor.setUserPayload(cpConfig.toUserPayload(new 
TezConfiguration(conf)));
{code}

 !Tez-CartesianProductSlowness.png! 




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21781) Vectorization: Incomplete vectorization of "or false" with CBO off

[Gopal V (JIRA)]

Gopal V created HIVE-21781:
--

 Summary: Vectorization: Incomplete vectorization of "or false" 
with CBO off
 Key: HIVE-21781
 URL: https://issues.apache.org/jira/browse/HIVE-21781
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Reporter: Gopal V


{code}
create temporary table foo (x int) ;
insert into foo values(1),(2),(3),(4),(5);
set hive.explain.user=false;

explain vectorization detail select count(case when (x=1 or false) then 1 else 
0 end ) from foo
{code}

{code}
| Group By Operator  |
|   aggregations: count(CASE WHEN (((x = 1) or false)) THEN 
(1) ELSE (0) END) |
|   Group By Vectorization:  |
|   aggregators: 
VectorUDAFCount(IfExprLongScalarLongScalar(col 3:boolean, val 1, val 
0)(children: VectorUDFAdaptor(((x = 1) or false))(children: 
LongColEqualLongScalar(col 0:int, val 1) -> 2:boolean) -> 3:boolean) -> 4:int) 
-> bigint |
|   className: VectorGroupByOperator |
{code}

The pass-through Calcite fixes this.

{code}
| OPTIMIZED SQL: SELECT COUNT(CASE WHEN `x` = 1 THEN 1 ELSE 0 END) AS `$f0` |
| FROM `default`.`foo` 
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21779) ACID: Delete deltas should not project the row struct

[Gopal V (JIRA)]

Gopal V created HIVE-21779:
--

 Summary: ACID: Delete deltas should not project the row struct
 Key: HIVE-21779
 URL: https://issues.apache.org/jira/browse/HIVE-21779
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.1.1, 4.0.0
Reporter: Gopal V


{code}
if (useDecimal64ColumnVector) {
  this.batch = deleteDeltaReader.getSchema().createRowBatchV2();
} else {
  this.batch = deleteDeltaReader.getSchema().createRowBatch();
}
{code}

Is creating a fully wide row-batch, despite the fact that all row columns have 
to be NULL in a delete delta.

createRowBatch() should follow includes and avoid over-allocating column vrbs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21768) JDBC: Strip the default union prefix for un-enclosed UNION queries

[Gopal V (JIRA)]

Gopal V created HIVE-21768:
--

 Summary: JDBC: Strip the default union prefix for un-enclosed 
UNION queries
 Key: HIVE-21768
 URL: https://issues.apache.org/jira/browse/HIVE-21768
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer
Affects Versions: 3.1.1, 4.0.0
Reporter: Gopal V


{code}
beeline> select 1  union all select 2;

+--+
| _u1._c0  |
+--+
| 2|
| 1|
+--+
{code}

The _u1 is superfluous and harmful to result schema parsing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21742) Vectorization: CASE result type casting

[Gopal V (JIRA)]

Gopal V created HIVE-21742:
--

 Summary: Vectorization: CASE result type casting
 Key: HIVE-21742
 URL: https://issues.apache.org/jira/browse/HIVE-21742
 Project: Hive
  Issue Type: Bug
  Components: Logical Optimizer, Vectorization
Affects Versions: 3.1.1
Reporter: Gopal V


{code}
create temporary table foo(q548284 int);
insert into foo values(1),(2),(3),(4),(5),(6);
select q548284, CASE WHEN ((q548284 = 1)) THEN (0.2) WHEN ((q548284 = 2)) THEN 
(0.4) WHEN ((q548284 = 3)) THEN (0.6) WHEN ((q548284 = 4)) THEN (0.8) WHEN 
((q548284 = 5)) THEN (1) ELSE (null) END from foo order by q548284 limit 1;
{code}

Fails with 

{code}
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to 
org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector
at 
org.apache.hadoop.hive.ql.exec.vector.DecimalColumnVector.setElement(DecimalColumnVector.java:130)
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprColumnNull.evaluate(IfExprColumnNull.java:101)
{code}

This gets fixed if the case return of (1) is turned into a (1.0).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21687) LLAP: LlapBaseInputFormat.getSplits(..., 0) should use LIMIT 0 query

[Gopal V (JIRA)]

Gopal V created HIVE-21687:
--

 Summary: LLAP: LlapBaseInputFormat.getSplits(..., 0) should use 
LIMIT 0 query
 Key: HIVE-21687
 URL: https://issues.apache.org/jira/browse/HIVE-21687
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


The 0 split query is useful in ensuring that the user has access to the tables 
and getting the result schema.

There's a fast-path for Tableau for "LIMIT 0" queries which can be used for 
this without touching the get_splits UDTF.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21646) Tez: Prevent TezTasks from escaping thread logging context

[Gopal V (JIRA)]

Gopal V created HIVE-21646:
--

 Summary: Tez: Prevent TezTasks from escaping thread logging context
 Key: HIVE-21646
 URL: https://issues.apache.org/jira/browse/HIVE-21646
 Project: Hive
  Issue Type: Bug
  Components: Tez
Reporter: Gopal V


If hive.exec.parallel is set to true to parallelize MoveTasks or StatsTasks, 
the Tez task does not benefit from a new thread and will lose all the thread 
context of the current query.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21548) Statistics: String statistics computation is slower with vectorization enabled

[Gopal V (JIRA)]

Gopal V created HIVE-21548:
--

 Summary: Statistics: String statistics computation is slower with 
vectorization enabled
 Key: HIVE-21548
 URL: https://issues.apache.org/jira/browse/HIVE-21548
 Project: Hive
  Issue Type: Improvement
  Components: Statistics, Vectorization
Reporter: Gopal V
 Attachments: string-stats-hll.png

Computing the hashCode of a String for HLL is going

byte[] -> Text  -> String -> byte[]

to get the Murmur3 hashCode of the byte[] region.

 !string-stats-hll.png! 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21547) Temp Tables: Use stORC format for temporary tables

[Gopal V (JIRA)]

Gopal V created HIVE-21547:
--

 Summary: Temp Tables: Use stORC format for temporary tables
 Key: HIVE-21547
 URL: https://issues.apache.org/jira/browse/HIVE-21547
 Project: Hive
  Issue Type: Improvement
  Components: ORC
Affects Versions: 4.0.0
Reporter: Gopal V


Using st(reaming)ORC (hive.exec.orc.delta.streaming.optimizations.enabled=true) 
format has massive performance advantages when creating data-sets which will 
not be stored for long-term.

The format is compatible with ORC for vectorization and other features, while 
being cheaper to write out to filesystem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3

[Gopal V (JIRA)]

Gopal V created HIVE-21531:
--

 Summary: Vectorization: all NULL hashcodes are not computed using 
Murmur3
 Key: HIVE-21531
 URL: https://issues.apache.org/jira/browse/HIVE-21531
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


The comments in Vectorized hash computation call out the MurmurHash 
implementation (the one using 0x5bd1e995), while the non-vectorized codepath 
calls out the Murmur3 one (using 0xcc9e2d51).

The comments here are wrong

{code}
 /**
   * Batch compute the hash codes for all the serialized keys.
   *
   * NOTE: MAJOR MAJOR ASSUMPTION:
   * We assume that HashCodeUtil.murmurHash produces the same result
   * as MurmurHash.hash with seed = 0 (the method used by 
ReduceSinkOperator for
   * UNIFORM distribution).
   */
  protected void computeSerializedHashCodes() {
int offset = 0;
int keyLength;
byte[] bytes = output.getData();
for (int i = 0; i < nonNullKeyCount; i++) {
  keyLength = serializedKeyLengths[i];
  hashCodes[i] = Murmur3.hash32(bytes, offset, keyLength, 0);
  offset += keyLength;
}
  }
{code}

but the wrong comment is followed in the Vector RS operator 

{code}
  System.arraycopy(nullKeyOutput.getData(), 0, nullBytes, 0, 
nullBytesLength);
  nullKeyHashCode = HashCodeUtil.calculateBytesHashCode(nullBytes, 0, 
nullBytesLength);
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21505) JDBC: Hive jdbc standalone does not have hadoop-common SASL servers

[Gopal V (JIRA)]

Gopal V created HIVE-21505:
--

 Summary: JDBC: Hive jdbc standalone does not have hadoop-common 
SASL servers
 Key: HIVE-21505
 URL: https://issues.apache.org/jira/browse/HIVE-21505
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Either the detection code should use SASL client classes or the shading should 
include it.

{code}
Caused by: java.lang.NoClassDefFoundError: 
org/apache/hadoop/security/SaslRpcServer
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge23.(HadoopThriftAuthBridge23.java:66)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge.getBridge(HadoopThriftAuthBridge.java:82)
at 
org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55)
at 
org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:588)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:337)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:224)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at org.notmysock.jdbc.JDBCActor.call(JDBCActor.java:49)
at org.notmysock.jdbc.JDBCActor.call(JDBCActor.java:15)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadP
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 70286: HIVE-21305: LLAP: Option to skip cache for ETL queries

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70286/#review213954
---


Ship it!




Ship It!

- Gopal V


On March 23, 2019, 2:47 a.m., prasanthj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70286/
> ---
> 
> (Updated March 23, 2019, 2:47 a.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-21305
> https://issues.apache.org/jira/browse/HIVE-21305
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21305: LLAP: Option to skip cache for ETL queries
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0c783e144d 
>   itests/src/test/resources/testconfiguration.properties 8c4d9b7de7 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 6252013335 
>   ql/src/test/queries/clientpositive/llap_io_etl.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/llap_io_etl.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70286/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> prasanthj
> 
>

[jira] [Created] (HIVE-21447) Constraints: Adding a constraint twice takes 120+ seconds to fail

[Gopal V (JIRA)]

Gopal V created HIVE-21447:
--

 Summary: Constraints: Adding a constraint twice takes 120+ seconds 
to fail
 Key: HIVE-21447
 URL: https://issues.apache.org/jira/browse/HIVE-21447
 Project: Hive
  Issue Type: Bug
  Components: Metastore, Standalone Metastore
Affects Versions: 4.0.0
Reporter: Gopal V


It reconnects to the metastore 24 times before failing without a real reason.

There's no useful error returned.

{code}
 alter table customer add constraint ${DB}_c_fsd foreign key  
(c_first_sales_date_sk) references date_dim (d_date_sk) disable novalidate rely;
INFO  : Compiling 
command(queryId=hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530): 
alter table customer add constraint tpcds_copy_orc_partitioned_1_c_fsd 
foreign key  (c_first_sales_date_sk) references date_dim (d_date_sk) disable 
novalidate rely
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling 
command(queryId=hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530); Time 
taken: 0.021 seconds
INFO  : Executing 
command(queryId=hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530): 
alter table customer add constraint tpcds_copy_orc_partitioned_1_c_fsd 
foreign key  (c_first_sales_date_sk) references date_dim (d_date_sk) disable 
novalidate rely
INFO  : Starting task [Stage-0:DDL] in serial mode
ERROR : FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. 
org.apache.thrift.TApplicationException: Internal error processing 
add_foreign_key
INFO  : Completed executing 
command(queryId=hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530); Time 
taken: 122.712 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 
from org.apache.hadoop.hive.ql.exec.DDLTask. 
org.apache.thrift.TApplicationException: Internal error processing 
add_foreign_key (state=08S01,code=1)
{code}

{code}
2019-03-14T06:12:57,299 WARN  [HiveServer2-Background-Pool: Thread-4295 
(hive_20190314061155_a060ca11-9547-4400-9564-9f0108c0e530)] 
metastore.RetryingMetaStoreClient: MetaStoreClientlost connection. Attempting 
to reconnect (13 of 24) after 5s. addForeignKey
org.apache.thrift.TApplicationException: Internal error processing 
add_foreign_key
at 
org.apache.thrift.TApplicationException.read(TApplicationException.java:111) 
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) 
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_add_foreign_key(ThriftHiveMetastore.java:1738)
 ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_foreign_key(ThriftHiveMetastore.java:1725)
 ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.addForeignKey(HiveMetaStoreClient.java:1033)
 ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
 ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy36.addForeignKey(Unknown Source) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_181]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_181]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3010)
 ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy36.addForeignKey(Unknown Source) ~[?:?]
at 
org.apache.hadoop.hive.ql.metadata.Hive.addForeignKey(Hive.java:5504) 
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.DDLTask.addConstraints(DDLTask.java:4080) 
~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21437) Vectorization: Decimal division with integer columns

[Gopal V (JIRA)]

Gopal V created HIVE-21437:
--

 Summary: Vectorization: Decimal division with integer columns
 Key: HIVE-21437
 URL: https://issues.apache.org/jira/browse/HIVE-21437
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Reporter: Gopal V


Vectorizer fails for

{code}
CREATE temporary TABLE `catalog_Sales`(
  `cs_quantity` int, 
  `cs_wholesale_cost` decimal(7,2), 
  `cs_list_price` decimal(7,2), 
  `cs_sales_price` decimal(7,2), 
  `cs_ext_discount_amt` decimal(7,2), 
  `cs_ext_sales_price` decimal(7,2), 
  `cs_ext_wholesale_cost` decimal(7,2), 
  `cs_ext_list_price` decimal(7,2), 
  `cs_ext_tax` decimal(7,2), 
  `cs_coupon_amt` decimal(7,2), 
  `cs_ext_ship_cost` decimal(7,2), 
  `cs_net_paid` decimal(7,2), 
  `cs_net_paid_inc_tax` decimal(7,2), 
  `cs_net_paid_inc_ship` decimal(7,2), 
  `cs_net_paid_inc_ship_tax` decimal(7,2), 
  `cs_net_profit` decimal(7,2))
 ;

explain vectorization detail select maxcs_ext_list_price - 
cs_ext_wholesale_cost) - cs_ext_discount_amt) + cs_ext_sales_price) / 2) from 
catalog_sales;
{code}

{code}
'Map Vectorization:'
'enabled: true'
'enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true'
'inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
'notVectorizedReason: SELECT operator: Could not instantiate 
DecimalColDivideDecimalScalar with arguments arguments: [21, 20, 22], argument 
classes: [Integer, Integer, Integer], exception: 
java.lang.IllegalArgumentException: java.lang.ClassCastException@63b56be0 stack 
trace: sun.reflect.GeneratedConstructorAccessor.newInstance(Unknown 
Source), 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45),
 java.lang.reflect.Constructor.newInstance(Constructor.java:423), 
org.apache.hadoop.hive.ql.exec.vector.VectorizationContext.instantiateExpression(VectorizationContext.java:2088),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4662),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.fixDecimalDataTypePhysicalVariations(Vectorizer.java:4602),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.vectorizeSelectOperator(Vectorizer.java:4584),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5171),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:923),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:809),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:776),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2400(Vectorizer.java:240),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:2038),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapOperators(Vectorizer.java:1990),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeMapWork(Vectorizer.java:1963),
 ...'
'vectorized: false'
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21417) JDBC: standalone jar relocates log4j interface classes

[Gopal V (JIRA)]

Gopal V created HIVE-21417:
--

 Summary: JDBC: standalone jar relocates log4j interface classes
 Key: HIVE-21417
 URL: https://issues.apache.org/jira/browse/HIVE-21417
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


The relocation of slf4j for ILoggerFactory breaks embedding in JVMs which have 
an slf4j impl locally.

org/apache/hive/org/slf4j/ILoggerFactory

Adding this jar to an existing slf4j env breaks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21400) Vectorization: LazyBinarySerializeWrite allocates Field() within the loop

[Gopal V (JIRA)]

Gopal V created HIVE-21400:
--

 Summary: Vectorization: LazyBinarySerializeWrite allocates Field() 
within the loop
 Key: HIVE-21400
 URL: https://issues.apache.org/jira/browse/HIVE-21400
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Affects Versions: 4.0.0
Reporter: Gopal V


GC thrash from an unexpected source in ReduceSinkOperator.

{code}
org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.resetWithoutOutput(LazyBinarySerializeWrite.java:136)
at 
org.apache.hadoop.hive.serde2.lazybinary.fast.LazyBinarySerializeWrite.reset(LazyBinarySerializeWrite.java:132)
at 
org.apache.hadoop.hive.ql.exec.vector.reducesink.VectorReduceSinkUniformHashOperator.process(VectorReduceSinkUniformHashOperator.java:180)
{code}

GC space is getting thrashed by the 

{code}
root = new Field(STRUCT);
{code}

for every row.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21368) Vectorization: Unnecessary Decimal64 -> HiveDecimal conversion

[Gopal V (JIRA)]

Gopal V created HIVE-21368:
--

 Summary: Vectorization: Unnecessary Decimal64 -> HiveDecimal 
conversion
 Key: HIVE-21368
 URL: https://issues.apache.org/jira/browse/HIVE-21368
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Joins projecting Decimal64 have a suspicious cast in the inner loop

{code}
ConvertDecimal64ToDecimal(col 14:decimal(7,2)/DECIMAL_64) -> 24:decimal(7,2)'
{code}

{code}
create temporary table foo(x int , y decimal(7,2));
create temporary table bar(x int , y decimal(7,2));
set hive.explain.user=false;
explain vectorization detail select sum(foo.y) from foo, bar where foo.x = 
bar.x;
{code}

{code}
'  Map Join Operator'
'condition map:'
' Inner Join 0 to 1'
'keys:'
'  0 _col0 (type: int)'
'  1 _col0 (type: int)'
'Map Join Vectorization:'
'bigTableKeyColumnNums: [0]'
'bigTableRetainedColumnNums: [3]'
'bigTableValueColumnNums: [3]'
'bigTableValueExpressions: 
ConvertDecimal64ToDecimal(col 1:decimal(7,2)/DECIMAL_64) -> 3:decimal(7,2)'
'className: VectorMapJoinInnerBigOnlyLongOperator'
'native: true'
'nativeConditionsMet: 
hive.mapjoin.optimized.hashtable IS true, 
hive.vectorized.execution.mapjoin.native.enabled IS true, hive.execution.engine 
tez IN [tez, spark] IS true, One MapJoin Condition IS true, No nullsafe IS 
true, Small table vectorizes IS true, Fast Hash Table and No Hybrid Hash Join 
IS true'
'projectedOutputColumnNums: [3]'
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21344) CBO: Materialized view registry is not used for Calcite planner

[Gopal V (JIRA)]

Gopal V created HIVE-21344:
--

 Summary: CBO: Materialized view registry is not used for Calcite 
planner
 Key: HIVE-21344
 URL: https://issues.apache.org/jira/browse/HIVE-21344
 Project: Hive
  Issue Type: Bug
  Components: Materialized views
Affects Versions: 4.0.0
Reporter: Gopal V
 Attachments: mv-get-from-remote.png

{code}
// This is not a rebuild, we retrieve all the materializations. In turn, we do 
not need
  // to force the materialization contents to be up-to-date, as this is 
not a rebuild, and
  // we apply the user parameters 
(HIVE_MATERIALIZED_VIEW_REWRITING_TIME_WINDOW) instead.
  materializations = 
db.getAllValidMaterializedViews(getTablesUsed(basePlan), false, getTxnMgr());
}
{code}

!mv-get-from-remote.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21343) CBO: CalcitePlanner debug logging is expensive and costly

[Gopal V (JIRA)]

Gopal V created HIVE-21343:
--

 Summary: CBO: CalcitePlanner debug logging is expensive and costly
 Key: HIVE-21343
 URL: https://issues.apache.org/jira/browse/HIVE-21343
 Project: Hive
  Issue Type: Bug
  Components: CBO
Affects Versions: 4.0.0
Reporter: Gopal V
 Attachments: Reloptutil-toString.png

{code}
  //Remove subquery
  LOG.debug("Plan before removing subquery:\n" + 
RelOptUtil.toString(calciteGenPlan));
  calciteGenPlan = hepPlan(calciteGenPlan, false, 
mdProvider.getMetadataProvider(), null,
  new HiveSubQueryRemoveRule(conf));
  LOG.debug("Plan just after removing subquery:\n" + 
RelOptUtil.toString(calciteGenPlan));

  calciteGenPlan = HiveRelDecorrelator.decorrelateQuery(calciteGenPlan);
  LOG.debug("Plan after decorrelation:\n" + 
RelOptUtil.toString(calciteGenPlan));
{code}

The LOG.debug() consumes more CPU than the actual planner steps.

 !Reloptutil-toString.png! 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21340) CBO: Prune non-key columns feeding into a SemiJoin

[Gopal V (JIRA)]

Gopal V created HIVE-21340:
--

 Summary: CBO: Prune non-key columns feeding into a SemiJoin
 Key: HIVE-21340
 URL: https://issues.apache.org/jira/browse/HIVE-21340
 Project: Hive
  Issue Type: Bug
  Components: CBO
Affects Versions: 4.0.0
Reporter: Gopal V


{code}
explain cbo 
with ss as 
(select count(1), ss_item_sk, ss_ticket_number from 
store_sales group by ss_item_sk, ss_ticket_number 
having count(1) > 1) 
select count(1) from item where i_item_sk IN (select ss_item_sk from ss);
{code}

Notice the {{HiveProject(ss_item_sk=[$0], ss_ticket_number=[$1], $f2=[$2])}} 

Only ss_item_sk is relevant for the HiveSemiJoin

{code}
CBO PLAN:
HiveAggregate(group=[{}], agg#0=[count()])
  HiveSemiJoin(condition=[=($0, $1)], joinType=[inner])
HiveProject(i_item_sk=[$0])
  HiveFilter(condition=[IS NOT NULL($0)])
HiveTableScan(table=[[tpcds_copy_orc_partitioned_1, item]], 
table:alias=[item])
HiveProject(ss_item_sk=[$0], ss_ticket_number=[$1], $f2=[$2])
  HiveFilter(condition=[>($2, 1)])
HiveAggregate(group=[{1, 8}], agg#0=[count()])
  HiveFilter(condition=[IS NOT NULL($1)])
HiveTableScan(table=[[tpcds_copy_orc_partitioned_1, 
store_sales]], table:alias=[store_sales])
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21339) LLAP: Cache hit also initializes an FS object

[Gopal V (JIRA)]

Gopal V created HIVE-21339:
--

 Summary: LLAP: Cache hit also initializes an FS object 
 Key: HIVE-21339
 URL: https://issues.apache.org/jira/browse/HIVE-21339
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 4.0.0
Reporter: Gopal V
 Attachments: llap-cache-fs-get.png

https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L214

{code}
// 1. Get file metadata from cache, or create the reader and read it.
// Don't cache the filesystem object for now; Tez closes it and FS cache 
will fix all that
fs = split.getPath().getFileSystem(jobConf);
fileKey = determineFileId(fs, split,
HiveConf.getBoolVar(daemonConf, 
ConfVars.LLAP_CACHE_ALLOW_SYNTHETIC_FILEID),
HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_CACHE_DEFAULT_FS_FILE_ID),
!HiveConf.getBoolVar(daemonConf, ConfVars.LLAP_IO_USE_FILEID_PATH)
);
{code}

 !llap-cache-fs-get.png! 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21294) Vectorization: 1-reducer Shuffle can skip the object hash functions

[Gopal V (JIRA)]

Gopal V created HIVE-21294:
--

 Summary: Vectorization: 1-reducer Shuffle can skip the object hash 
functions
 Key: HIVE-21294
 URL: https://issues.apache.org/jira/browse/HIVE-21294
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Reporter: Gopal V


VectorObjectSinkHashOperator can skip the object hashing entirely if the 
reducer count = 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21268) REPL: Repl dump can output - Database, Table, Dir, last_repl_id

[Gopal V (JIRA)]

Gopal V created HIVE-21268:
--

 Summary: REPL: Repl dump can output - Database, Table, Dir, 
last_repl_id
 Key: HIVE-21268
 URL: https://issues.apache.org/jira/browse/HIVE-21268
 Project: Hive
  Issue Type: Improvement
Reporter: Gopal V


{code}
INFO  : Completed executing 
command(queryId=root_20190214061031_639e3a52-5c62-40be-a3cd-3e0b18b7b41d); Time 
taken: 0.374 seconds
INFO  : OK
++---+
|  dump_dir  | last_repl_id  |
++---+
| /user/root/repl/a74389d0-7cde-4cf4-aa40-3079a98b80a8 | 1104594   |
++---+
1 row selected (0.445 seconds)
{code}

is somewhat hard to associate back to the table name.

The logs a couple of lines above actually print the operation detail.

{code}
INFO  : REPL::TABLE_DUMP: 
{"dbName":"tpcds_bin_partitioned_orc_1000","tableName":"item","tableType":"MANAGED_TABLE","tablesDumpProgress":"1/38","dumpTime":1550124632}
{code}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21233) ACID: Fix build warnings from HIVE-20699

[Gopal V (JIRA)]

Gopal V created HIVE-21233:
--

 Summary: ACID: Fix build warnings from HIVE-20699
 Key: HIVE-21233
 URL: https://issues.apache.org/jira/browse/HIVE-21233
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 4.0.0
Reporter: Gopal V


{code}
Lines that start with ? in the ASF License  report indicate files that do 
not have an Apache license header:
 !? 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15992/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFValidateAcidSortOrder.java
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21232) LLAP: Add a cache-miss friendly split affinity provider

[Gopal V (JIRA)]

Gopal V created HIVE-21232:
--

 Summary: LLAP: Add a cache-miss friendly split affinity provider
 Key: HIVE-21232
 URL: https://issues.apache.org/jira/browse/HIVE-21232
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


If one of the LLAP nodes have data-locality, preferring that over another does 
have advantages for the first query or a more general cache-miss.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21229) Tez: Provide a Node-Label split affinity provider

[Gopal V (JIRA)]

Gopal V created HIVE-21229:
--

 Summary: Tez: Provide a Node-Label split affinity provider
 Key: HIVE-21229
 URL: https://issues.apache.org/jira/browse/HIVE-21229
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


When the queue has node-labels, the delays involved in waiting for node 
locality are a net-loss to the query.

Asking for locality in a node which is not available to the current queue is 
slowing down YARN task allocations.

Also specifically, for the 3-replica HDFS case, picking the replica that 
belongs within the node-label is more efficient than picking a non-available 
node & falling back to rack_local instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally

[Gopal V (JIRA)]

Gopal V created HIVE-21225:
--

 Summary: ACID: getAcidState() should cache a recursive dir listing 
locally
 Key: HIVE-21225
 URL: https://issues.apache.org/jira/browse/HIVE-21225
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Gopal V


Currently getAcidState() makes 3 calls into the FS api which could be answered 
by making a single recursive listDir call and reusing the same data to check 
for isRawFormat() and isValidBase().

All delta operations for a single partition can go against a single listed 
directory snapshot instead of interacting with the NameNode or ObjectStore 
within the inner loop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21012) CBO: CASE + CONCAT expr is not converted to optimized SQL

[Gopal V (JIRA)]

Gopal V created HIVE-21012:
--

 Summary: CBO: CASE + CONCAT expr is not converted to optimized SQL
 Key: HIVE-21012
 URL: https://issues.apache.org/jira/browse/HIVE-21012
 Project: Hive
  Issue Type: Bug
  Components: CBO
Affects Versions: 4.0.0
Reporter: Gopal V


{code}
create temporary table t3 (receipt_required_flag string, 
inspection_required_flag string);

explain extended select  CASE
WHEN concat(concat(coalesce(t3.receipt_required_flag, 'N'), '~'), 
coalesce(t3.inspection_required_flag, 'N')) = 'N~N' THEN '2-Way Match'
WHEN concat(concat(coalesce(t3.receipt_required_flag, 'N'), '~'), 
coalesce(t3.inspection_required_flag, 'N')) = 'Y~N' THEN '3-Way Match'
WHEN concat(concat(coalesce(t3.receipt_required_flag, 'N'), '~'), 
coalesce(t3.inspection_required_flag, 'N')) = 'Y~Y' THEN '4-Way Match'
END from t3;
{code}

does not print an optimized SQL because

{code}
2018-12-05T22:52:26,193  WARN [HiveServer2-Background-Pool: Thread-2100] 
parse.CalcitePlanner: Rel2SQL Rewrite threw error
java.lang.UnsupportedOperationException: class 
org.apache.calcite.sql.SqlSyntax$6: SPECIAL
at org.apache.calcite.util.Util.needToImplement(Util.java:927) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlSyntax$6.unparse(SqlSyntax.java:116) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlOperator.unparse(SqlOperator.java:332) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:333) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlCall.unparse(SqlCall.java:103) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlUtil.unparseBinarySyntax(SqlUtil.java:323) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlSyntax$3.unparse(SqlSyntax.java:65) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlOperator.unparse(SqlOperator.java:332) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:333) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlCall.unparse(SqlCall.java:103) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at 
org.apache.calcite.sql.fun.SqlCaseOperator.unparse(SqlCaseOperator.java:306) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:333) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlCall.unparse(SqlCall.java:103) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlAsOperator.unparse(SqlAsOperator.java:76) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlDialect.unparseCall(SqlDialect.java:333) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlCall.unparse(SqlCall.java:103) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlNodeList.commaList(SqlNodeList.java:121) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at 
org.apache.calcite.sql.SqlOperator.unparseListClause(SqlOperator.java:349) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at 
org.apache.calcite.sql.SqlOperator.unparseListClause(SqlOperator.java:338) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at 
org.apache.calcite.sql.SqlSelectOperator.unparse(SqlSelectOperator.java:152) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlSelect.unparse(SqlSelect.java:240) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:152) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at org.apache.calcite.sql.SqlNode.toSqlString(SqlNode.java:158) 
~[calcite-core-1.16.0.3.1.0.0-SNAPSHOT.jar:1.16.0.3.1.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedSql(CalcitePlanner.java:1477)
 ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21005) LLAP: Reading more stripes per-split leaks ZlibCodecs

[Gopal V (JIRA)]

Gopal V created HIVE-21005:
--

 Summary: LLAP: Reading more stripes per-split leaks ZlibCodecs
 Key: HIVE-21005
 URL: https://issues.apache.org/jira/browse/HIVE-21005
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


OrcEncodedDataReader - calls ensureDataReader in a loop, overwriting itself

{code}
for (int stripeIxMod = 0; stripeIxMod < stripeRgs.length; ++stripeIxMod) {

// 6.2. Ensure we have stripe metadata. We might have read it before 
for RG filtering.
if (stripeMetadatas != null) {
  stripeMetadata = stripeMetadatas.get(stripeIxMod);
} else {
...
  ensureDataReader();
...
}
{code}

{code}
  private void ensureDataReader() throws IOException {
...
stripeReader = orcReader.encodedReader(
fileKey, dw, dw, useObjectPools ? POOL_FACTORY : null, trace, 
useCodecPool, cacheTag);
{code}

creates new encodedReader without closing previous stripe's encoded reader.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20999) LLAP IO: MutableQuantiles is contended heavily

[Gopal V (JIRA)]

Gopal V created HIVE-20999:
--

 Summary: LLAP IO: MutableQuantiles is contended heavily
 Key: HIVE-20999
 URL: https://issues.apache.org/jira/browse/HIVE-20999
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 3.1.1
Reporter: Gopal V


MutableQuantiles::add() is synchronized across all threads.

{code}
IO-Elevator-Thread-0 [DAEMON] State: BLOCKED CPU usage on sample: 316ms
org.apache.hadoop.metrics2.lib.MutableQuantiles.add(long) 
MutableQuantiles.java:133
org.apache.hadoop.hive.llap.metrics.LlapDaemonIOMetrics.addDecodeBatchTime(long)
 LlapDaemonIOMetrics.java:98
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedColumnBatch)
 EncodedDataConsumer.java:89
org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(Object) 
EncodedDataConsumer.java:34
org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(int,
 StripeInformation, OrcProto$RowIndex[], List, List, boolean[], boolean[], 
Consumer) EncodedReaderImpl.java:530
org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead() 
OrcEncodedDataReader.java:407
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20989) JDBC: The GetOperationStatus + log can block query progress via sleep()

[Gopal V (JIRA)]

Gopal V created HIVE-20989:
--

 Summary: JDBC: The GetOperationStatus + log can block query 
progress via sleep()
 Key: HIVE-20989
 URL: https://issues.apache.org/jira/browse/HIVE-20989
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


There is an exponential sleep operation inside the CLIService which can end up 
adding tens of seconds to a query which has already completed.

{code}
"HiveServer2-Handler-Pool: Thread-9373" #9373 prio=5 os_prio=0 
tid=0x7f4d5e72d800 nid=0xb634a waiting on condition [0x7f28d06a5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hive.service.cli.CLIService.progressUpdateLog(CLIService.java:506)
at 
org.apache.hive.service.cli.CLIService.getOperationStatus(CLIService.java:480)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.GetOperationStatus(ThriftCLIService.java:695)
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1757)
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1742)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}

The sleep loop is on the server side.

{code}
private static final long PROGRESS_MAX_WAIT_NS = 30 * 10l;

private JobProgressUpdate progressUpdateLog(boolean isProgressLogRequested, 
Operation operation, HiveConf conf) {
...
long startTime = System.nanoTime();
int timeOutMs = 8;
try {
  while (sessionState.getProgressMonitor() == null && !operation.isDone()) {
long remainingMs = (PROGRESS_MAX_WAIT_NS - (System.nanoTime() - 
startTime)) / 100l;
if (remainingMs <= 0) {
  LOG.debug("timed out and hence returning progress log as NULL");
  return new JobProgressUpdate(ProgressMonitor.NULL);
}
Thread.sleep(Math.min(remainingMs, timeOutMs));
timeOutMs <<= 1;
  }
{code}

After about 16 seconds of execution of the query, the timeOutMs is 16384 ms, 
which means the next sleep cycle is for min(30 - 17, 16) = 13.

If the query finishes on the 17th second, the JDBC server will only respond 
after the 30th second when it will check for operation.isDone() and return.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20983) Vectorization: Scale up small hashtables, when collisions are detected

[Gopal V (JIRA)]

Gopal V created HIVE-20983:
--

 Summary: Vectorization: Scale up small hashtables, when collisions 
are detected
 Key: HIVE-20983
 URL: https://issues.apache.org/jira/browse/HIVE-20983
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Hive's hashtable estimates are getting better with HyperLogLog stats in place, 
but an accurate estimate does not always result in a low number of collisions.

The hashtables which contain a very small number of items tend to lose their 
O(1) lookup performance where there are collisions. Since collisions are easy 
to detect within the fast hashtable implementation, a rehashing to a higher 
size will help these small hashtables avoid collisions and go back to O(1) perf.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20973) Optimizer: Reduce de-dup changes the hash-function of a reducer edge

[Gopal V (JIRA)]

Gopal V created HIVE-20973:
--

 Summary: Optimizer: Reduce de-dup changes the hash-function of a 
reducer edge
 Key: HIVE-20973
 URL: https://issues.apache.org/jira/browse/HIVE-20973
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


{code}
private static void 
propagateMaxNumReducers(ReduceSinkJoinDeDuplicateProcCtx dedupCtx,
ReduceSinkOperator rsOp, int maxNumReducers) throws 
SemanticException {
  if (rsOp == null) {
// Bail out
return;
  }
  if (rsOp.getChildOperators().get(0) instanceof MapJoinOperator ||
  rsOp.getChildOperators().get(0) instanceof 
CommonMergeJoinOperator) {
for (Operator p : 
rsOp.getChildOperators().get(0).getParentOperators()) {
  ReduceSinkOperator pRSOp = (ReduceSinkOperator) p;
  pRSOp.getConf().setReducerTraits(EnumSet.of(ReducerTraits.FIXED));
  pRSOp.getConf().setNumReducers(maxNumReducers);
  LOG.debug("Set {} to FIXED parallelism: {}", pRSOp, maxNumReducers);
  if (pRSOp.getConf().isForwarding()) {
ReduceSinkOperator newRSOp =
CorrelationUtilities.findFirstPossibleParent(
pRSOp, ReduceSinkOperator.class, dedupCtx.trustScript());
propagateMaxNumReducers(dedupCtx, newRSOp, maxNumReducers);
  }
}
{code}

FIXED used to mean AUTOPARALLEL=false, but now FIXED means a different hash 
function from UNIFORM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20951) LLAP: Set Xms to 50% always

[Gopal V (JIRA)]

Gopal V created HIVE-20951:
--

 Summary: LLAP: Set Xms to 50% always 
 Key: HIVE-20951
 URL: https://issues.apache.org/jira/browse/HIVE-20951
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 3.1.1, 4.0.0
Reporter: Gopal V


The lack of GC pauses is killing LLAP containers whenever the significant 
amount of memory is consumed by the off-heap structures which aren't cleaned up 
automatically until the GC runs.

There's a java.nio.DirectByteBuffer.Deallocator which runs when the Direct 
buffers are garbage collected, which actually does the cleanup of the 
underlying off-heap buffers.

The lack of Garbage collection activity for several hours while responding to 
queries triggers a build-up of these off-heap structures which end up forcing 
YARN to kill the process instead.

It is better to hit a GC pause occasionally rather than to lose a node every 
few hours.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20888) TxnHandler: sort() called on immutable lists

[Gopal V (JIRA)]

Gopal V created HIVE-20888:
--

 Summary: TxnHandler: sort() called on immutable lists
 Key: HIVE-20888
 URL: https://issues.apache.org/jira/browse/HIVE-20888
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


{code}
} else {
  assert (!rqst.isSetSrcTxnToWriteIdList());
  assert (rqst.isSetTxnIds());
  txnIds = rqst.getTxnIds();
}

Collections.sort(txnIds); //easier to read logs and for assumption done 
in replication flow
{code}

when the input comes from

{code}
  @Override
  public long allocateTableWriteId(long txnId, String dbName, String tableName) 
throws TException {
return allocateTableWriteIdsBatch(Collections.singletonList(txnId), dbName, 
tableName).get(0).getWriteId();
  }
{code}

{code}
java.lang.UnsupportedOperationException: null
at java.util.AbstractList.set(AbstractList.java:132) ~[?:1.8.0]
at java.util.AbstractList$ListItr.set(AbstractList.java:426) ~[?:1.8.0]
at java.util.Collections.sort(Collections.java:170) ~[?:1.8.0]
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.allocateTableWriteIds(TxnHandler.java:1523)
 ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.allocate_table_write_ids(HiveMetaStore.java:7349)
 ~[hive-standalone-metastore-server-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
{code}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20887) Tests: openjdk 8 has a bug that prevents surefire from working

[Gopal V (JIRA)]

Gopal V created HIVE-20887:
--

 Summary: Tests: openjdk 8 has a bug that prevents surefire from 
working
 Key: HIVE-20887
 URL: https://issues.apache.org/jira/browse/HIVE-20887
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


It looks like the problem is https://bugs.openjdk.java.net/browse/JDK-8030046. 
It looks like:

{code:bash}
[ERROR] Caused by: 
org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
terminated without properly saying goodbye. VM crash or System.exit called?
{code}

The surefire-reports/*.dumpstream looks like:
{code:bash}
Error: Could not find or load main class 
org.apache.maven.surefire.booter.ForkedBooter
{code}

 and we can work around the problem by changing the surefire configuration:

{code:bash}
+  false
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20890) ACID: Allow whole table ReadLocks to skip all partition locks

[Gopal V (JIRA)]

Gopal V created HIVE-20890:
--

 Summary: ACID: Allow whole table ReadLocks to skip all partition 
locks
 Key: HIVE-20890
 URL: https://issues.apache.org/jira/browse/HIVE-20890
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Gopal V


HIVE-19369 proposes adding a EXCL_WRITE lock which does not wait for any 
SHARED_READ locks for insert operations - in the presence of that lock, the 
insert overwrite no longer takes an exclusive lock.

The only exclusive operation will be a schema change or drop table, which 
should take an exclusive lock on the entire table directly.

{code}
explain locks select * from tpcds_bin_partitioned_orc_1000.store_sales where 
ss_sold_date_sk=2452626 

++
|  Explain   |
++
| LOCK INFORMATION:  |
| tpcds_bin_partitioned_orc_1000.store_sales -> SHARED_READ |
| tpcds_bin_partitioned_orc_1000.store_sales.ss_sold_date_sk=2452626 -> 
SHARED_READ |
++
{code}

So the per-partition SHARED_READ locks are no longer necessary, if the lock 
builder already includes the table-wide SHARED_READ locks.

The removal of entire partitions is the only part which needs to be taken care 
of within this semantics as row-removal instead of directory removal (i.e "drop 
partition" -> "truncate partition" and have the truncation trigger a whole 
directory cleaner, so that the partition disappears when there are 0 rows left).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20854) Sensible Defaults: Hive's Zookeeper heartbeat interval is 20 minutes, change to 2

[Gopal V (JIRA)]

Gopal V created HIVE-20854:
--

 Summary: Sensible Defaults: Hive's Zookeeper heartbeat interval is 
20 minutes, change to 2
 Key: HIVE-20854
 URL: https://issues.apache.org/jira/browse/HIVE-20854
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


{code}
HIVE_ZOOKEEPER_SESSION_TIMEOUT("hive.zookeeper.session.timeout", 
"120ms",
new TimeValidator(TimeUnit.MILLISECONDS),
"ZooKeeper client's session timeout (in milliseconds). The client is 
disconnected, and as a result, all locks released, \n" +
"if a heartbeat is not sent in the timeout."),
{code}

That's 1,200,000ms which is too long for all practical purposes - a 20 minute 
outage in case a node has a failure is too long.

That is too long for the JDBC load-balancing, LLAP failure tolerance and the 
lock manager expiry.

Change to 2 minutes, as a sensible default



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20802) Statistics: Provide per-column statistics expanded for "explain formatted"

[Gopal V (JIRA)]

Gopal V created HIVE-20802:
--

 Summary: Statistics: Provide per-column statistics expanded for 
"explain formatted"
 Key: HIVE-20802
 URL: https://issues.apache.org/jira/browse/HIVE-20802
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Debugging the internal per-column statistics of Hive is challenging, 
particularly when the estimations are approximate (hive.stats.estimate=true).

Adding an extra columnStatistics field for all operators in "explain formatted" 
will allow tools to analyze the internal statistics transforms done by each 
operator without having to refer to debug logs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking

[Gopal V (JIRA)]

Gopal V created HIVE-20801:
--

 Summary: ACID: Allow DbTxnManager to ignore non-ACID table locking
 Key: HIVE-20801
 URL: https://issues.apache.org/jira/browse/HIVE-20801
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Enabling ACIDv1 on a cluster produces a central locking bottleneck for all 
table types, which is not always the intention.

The Hive locking for non-acid tables are advisory (i.e a client can write/read 
without locking), which means that the implementation does not offer strong 
consistency despite the lock manager consuming resources centrally.

Disabling this lock acquisition would improve the performance of non-ACID 
tables co-existing with a globally configured DbTxnManager implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20764) Tez: ReduceRecordProcessor::init() can avoid doing .waitForAllInputsReady()

[Gopal V (JIRA)]

Gopal V created HIVE-20764:
--

 Summary: Tez: ReduceRecordProcessor::init() can avoid doing 
.waitForAllInputsReady()
 Key: HIVE-20764
 URL: https://issues.apache.org/jira/browse/HIVE-20764
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


The ReduceRecordProcessor waits for all inputs to be ready before starting to 
initialize the input operators.

This is not necessary for the unsorted shuffle cases, as just having one input 
(the hash side) entirely ready is sufficient to start doing useful work with 
the operators (i.e build the hashtable).

{code}
if (shuffleInputs != null) {
  l4j.info("Waiting for ShuffleInputs to become ready");
  processorContext.waitForAllInputsReady(new 
ArrayList(shuffleInputs));
}
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20758) Constraints: Show create table does not show constraints

[Gopal V (JIRA)]

Gopal V created HIVE-20758:
--

 Summary: Constraints: Show create table does not show constraints
 Key: HIVE-20758
 URL: https://issues.apache.org/jira/browse/HIVE-20758
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Even though the desc formatted shows the constraints, the show create table 
does not

{code}
| # Primary Key  | NULL 
  | NULL   |
| Table: | 
tpcds_bin_partitioned_orc_1.inventory  | NULL   
|
| Constraint Name:   | pk_in
  | NULL   |
| Column Names:  | inv_date_sk  
  | inv_item_sk|
|| NULL 
  | NULL   |
| # Foreign Keys | NULL 
  | NULL   |
| Table: | 
tpcds_bin_partitioned_orc_1.inventory  | NULL   
|
| Constraint Name:   | inv_d
  | NULL   |
| Parent Column Name:tpcds_bin_partitioned_orc_1.date_dim.d_date_sk | 
Column Name:inv_date_sk| Key Sequence:1 
|
|| NULL 
  | NULL   |
| Constraint Name:   | inv_i
  | NULL   |
| Parent Column Name:tpcds_bin_partitioned_orc_1.item.i_item_sk | Column 
Name:inv_item_sk| Key Sequence:1
 |
|| NULL 
  | NULL   |
| Constraint Name:   | inv_w
  | NULL   |
| Parent Column Name:tpcds_bin_partitioned_orc_1.warehouse.w_warehouse_sk | 
Column Name:inv_warehouse_sk   | Key Sequence:1 
|
|| NULL 
  | NULL   |
{code}

But 

{code}
++
|   createtab_stmt   |
++
| CREATE TABLE `inventory`(  |
|   `inv_item_sk` bigint,|
|   `inv_warehouse_sk` bigint,   |
|   `inv_quantity_on_hand` int,  |
|   `inv_date_sk` bigint)|
| ROW FORMAT SERDE   |
|   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'  |
| STORED AS INPUTFORMAT  |
|   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'  |
| OUTPUTFORMAT   |
|   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
| LOCATION   |
|   
'hdfs:///warehouse/tablespace/managed/hive/tpcds_bin_partitioned_orc_1.db/inventory'
 |
| TBLPROPERTIES (|
|   'bucketing_version'='2', |
|   'transactional'='true',  |
|   'transactional_properties'='default',|
|   'transient_lastDdlTime'='1539710410')|
++
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20749) Constraints: Null scan optimizer does not remove entire TableScan

[Gopal V (JIRA)]

Gopal V created HIVE-20749:
--

 Summary: Constraints: Null scan optimizer does not remove entire 
TableScan
 Key: HIVE-20749
 URL: https://issues.apache.org/jira/browse/HIVE-20749
 Project: Hive
  Issue Type: Bug
  Components: CBO, Logical Optimizer
Reporter: Gopal V


With PK constraints on store table, the following query fails to remove the 
entire join (and cut the query short).

{code}
 explain select count(1) from store_sales, store where ss_store_sk = s_store_sk 
and s_store_sk is null and ss_sold_date_sk between 2450816 and 2450816+365;
{code}

Calcite generates the following optimized query, but the NullScan optimizer 
does not kick in for the query.

{code}
SELECT COUNT(*) AS `$f0`
FROM (SELECT *
FROM (SELECT 0 AS `DUMMY`
FROM `tpcds_bin_partitioned_orc_1`.`store_sales`) AS `t`
WHERE FALSE) AS `t0`,
(SELECT *
FROM (SELECT 0 AS `DUMMY`
FROM `tpcds_bin_partitioned_orc_1`.`store`) AS `t1`
WHERE FALSE) AS `t2`
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20739) CBO: Rewrite INTERSECTS queries as EXISTS semi-joins

[Gopal V (JIRA)]

Gopal V created HIVE-20739:
--

 Summary: CBO: Rewrite INTERSECTS queries as EXISTS semi-joins
 Key: HIVE-20739
 URL: https://issues.apache.org/jira/browse/HIVE-20739
 Project: Hive
  Issue Type: Improvement
Reporter: Gopal V


INTERSECTS clause currently materializes both sides before removing entries 
from each other.

This means that if the intersects has filters on one side which apply to the 
other side, it is not transitively pushed to the other side.

Here's the snippet from the TPC-DS Query8 where that is visible.

{code}
SELECT substr(ca_zip,1,5) ca_zip
  FROM customer_address
  WHERE substr(ca_zip,1,5) IN (
  '89436', /* ... */ ','15492'))
 intersect
 (select ca_zip
  from (SELECT substr(ca_zip,1,5) ca_zip,count(*) cnt
FROM customer_address, customer
WHERE ca_address_sk = c_current_addr_sk and
  c_preferred_cust_flag='Y'
group by ca_zip
having count(*) > 10)A1))A2
{code}

https://github.com/hortonworks/hive-testbench/blob/hdp3/sample-queries-tpcds/query8.sql#L92

where the entire join output of (customer_address x customer) is produced where 
the transitive filter inference could be applied to push the filter to the 
customer_address in the 2nd set in intersection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20624) Vectorization: Fast Hash table should not double after certain size, instead should grow

[Gopal V (JIRA)]

Gopal V created HIVE-20624:
--

 Summary: Vectorization: Fast Hash table should not double after 
certain size, instead should grow
 Key: HIVE-20624
 URL: https://issues.apache.org/jira/browse/HIVE-20624
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Reporter: Gopal V


The reason to use Power of two is to simplify the inner loop for the hash 
function, but this is a significant memory issue when dealing with somewhat 
larger hashtables like the customer or customer_address hashtables in TPC-DS.

This doubling is pretty bad when the hashtable load-factor is 0.75 and the 
expected key count is 65M (for customer/customer_address)

{code}
long worstCaseNeededSlots = 1L << DoubleMath.log2(numRows / 
hashTableLoadFactor, RoundingMode.UP);
{code}

That estimate is actually matching the actual impl, but after acquiring 65M 
items in a single array, the rehashing will require a temporary growth to 
65+128 while the rehash is in progress, all to fit exactly 65 back into it.

Fixing the estimate to match the implementation produced a number of 
regressions in query runtimes, though the part that needs fixing is the 
doubling implementation.

The obvious solution is to add 4M more everytime and use a modulo function or 
the Lemire's multiply + shift operation[1], but more on that in comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20623) Shared work: Extend sharing of map-join cache entries in LLAP

[Gopal V (JIRA)]

Gopal V created HIVE-20623:
--

 Summary: Shared work: Extend sharing of map-join cache entries in 
LLAP
 Key: HIVE-20623
 URL: https://issues.apache.org/jira/browse/HIVE-20623
 Project: Hive
  Issue Type: Improvement
  Components: llap, Logical Optimizer
Reporter: Gopal V
 Attachments: hash-shared-work.svg

For a query like this

{code}
with all_sales as (
select ss_customer_sk as customer_sk, ss_ext_list_price-ss_ext_discount_amt as 
ext_price from store_sales
UNION ALL
select ws_bill_customer_sk as customer_sk, 
ws_ext_list_price-ws_ext_discount_amt as ext_price from web_sales
UNION ALL
select cs_bill_customer_sk as customer_sk, cs_ext_sales_price - 
cs_ext_discount_amt as ext_price from catalog_sales)
select sum(ext_price) total_price, c_customer_id from all_sales, customer 
where customer_sk = c_customer_sk
group by c_customer_id
order by total_price desc 
limit 100;
{code}

The hashtable used for all 3 joins are identical, which is loaded 3x times in 
the same LLAP instance because they are named.

{code}
cacheKey = "HASH_MAP_" + this.getOperatorId() + "_container";
{code}

in the cache.

If those are identical in nature (i.e vectorization, hashtable type etc), then 
the duplication is just wasted CPU, memory and network - using the cache name 
for hashtables which will be identical in layout would be extremely useful.

In cases where the join is pushed through a UNION, those are identical.

This optimization can only be done without concern for accidental delays when 
the same upstream task is generating all of these hashtables, which is what is 
achieved by the shared scan optimizer already.

In case the shared work is not present, this has potential downsides - in case 
two customer broadcasts were sourced from "Map 1" and "Map 2", the Map 1 
builder will block the other task from reading from Map 2, even though Map 2 
might have started after, but finished ahead of Map 1.

So this specific optimization can always be considered for cases where the 
shared work unifies the operator tree and the parents of all the RS entries 
involved are same (& the RS layout is the same).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20622) Backwards Compat: Support non-zero padded date and timestamp entries

[Gopal V (JIRA)]

Gopal V created HIVE-20622:
--

 Summary: Backwards Compat: Support non-zero padded date and 
timestamp entries
 Key: HIVE-20622
 URL: https://issues.apache.org/jira/browse/HIVE-20622
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


The Date spec in Hive says

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-date

{noformat}
DATE values describe a particular year/month/day, in the form -­MM-­DD. For 
example, DATE '2013-­01-­01'
{noformat}

Even though the Hive date & timestamp spec are strict on their format 
specification, Hive has previously supported " -[m]m-[d]d" formatted dates 
and it is better to expand the spec rather than drop this compatibility.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20557) Constant Folding: null > (expr) is not constant folded to NULL

[Gopal V (JIRA)]

Gopal V created HIVE-20557:
--

 Summary: Constant Folding: null > (expr) is not constant folded to 
NULL
 Key: HIVE-20557
 URL: https://issues.apache.org/jira/browse/HIVE-20557
 Project: Hive
  Issue Type: Improvement
Reporter: Gopal V


{code}

create temporary table foo(col1 int, col2 int, col3 int, col5 int, col9 int) 
stored as orc;

explain select count(1) from foo where 
CASE WHEN (col5 is not null) THEN (((col1 / col5) > (col9 / col3))) ELSE ((null 
> (col9 / col3))) END;

explain select count(1), max((col9/col3) > null) from foo where 
CASE WHEN (col5 is not null) THEN (((col1 / col5) > (col9 / col3))) ELSE 
(((col9 / col3) > null)) END;
{code}


There's no sane way to write this query, but this is actually generated by CBO 
for an expression which looks like 

{code}
case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / 
t_w_firstyear.year_total else null end
   > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total 
/ t_s_firstyear.year_total else null end
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20509) Plan: fix wasted memory in plans with large partition counts

[Gopal V (JIRA)]

Gopal V created HIVE-20509:
--

 Summary: Plan: fix wasted memory in plans with large partition 
counts
 Key: HIVE-20509
 URL: https://issues.apache.org/jira/browse/HIVE-20509
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Reporter: Gopal V


{code}
  public void addPathToAlias(Path path, String newAlias){
ArrayList aliases = pathToAliases.get(path);
if (aliases == null) {
  aliases = new ArrayList<>();
  StringInternUtils.internUriStringsInPath(path);
  pathToAliases.put(path, aliases);
}
aliases.add(newAlias.intern());
  }
{code}

ArrayList::DEFAULT_CAPACITY is 10, so this wastes 500 bytes of memory due to 
the {{new ArrayList<>();}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20501) Vectorization: Closed range fast-path for Fast Long hashset

[Gopal V (JIRA)]

Gopal V created HIVE-20501:
--

 Summary: Vectorization: Closed range fast-path for Fast Long 
hashset 
 Key: HIVE-20501
 URL: https://issues.apache.org/jira/browse/HIVE-20501
 Project: Hive
  Issue Type: Improvement
Reporter: Gopal V


In scenarios where the surrogate keys are entirely contiguous, the cache can 
offer a fast-path for [min,max], without a further lookup in the hashtable.

{code}
hive> select min(c_customer_sk), max(c_customer_sk), max(c_customer_sk) - 
min(c_customer_sk), count(1) from customer;

1   650064996500
{code}






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20490) UDAF: Add an `approx_distinct` to Hive

[Gopal V (JIRA)]

Gopal V created HIVE-20490:
--

 Summary: UDAF: Add an `approx_distinct` to Hive
 Key: HIVE-20490
 URL: https://issues.apache.org/jira/browse/HIVE-20490
 Project: Hive
  Issue Type: Improvement
  Components: UDF
Affects Versions: 4.0.0
Reporter: Gopal V


This gives a direct pathway to call the equivalent of Hive's default 
compute_stats() UDAF which computes a HyperLogLog bitVector + approx distinct 
on a column.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20486) Kafka: Use Row SerDe + vectorization

[Gopal V (JIRA)]

Gopal V created HIVE-20486:
--

 Summary: Kafka: Use Row SerDe + vectorization
 Key: HIVE-20486
 URL: https://issues.apache.org/jira/browse/HIVE-20486
 Project: Hive
  Issue Type: Improvement
Reporter: Gopal V


KafkaHandler returns unvectorized rows which causes the operators downstream to 
be slower and sub-optimal.

Hive has a vectorization shim which allows Kafka streams without complex 
projections to be wrapped into a vectorized reader via 
{{hive.vectorized.use.row.serde.deserialize}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20455) Log spew from security.authorization.PrivilegeSynchonizer.run

[Gopal V (JIRA)]

Gopal V created HIVE-20455:
--

 Summary: Log spew from 
security.authorization.PrivilegeSynchonizer.run 
 Key: HIVE-20455
 URL: https://issues.apache.org/jira/browse/HIVE-20455
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Typo in the class name aside, this keeps running and thrashing through every 
single DB in the cluster if not all directories are actually accessible to the 
user who runs HS2.

{code}
2018-08-24T03:14:41,932  INFO [PrivilegeSynchonizer] metastore.HiveMetaStore: 
1: get_table : tbl=hive.airline_ontime.flights_raw
2018-08-24T03:14:41,932  INFO [PrivilegeSynchonizer] HiveMetaStore.audit: 
ugi=gopal ip=unknown-ip-addr  cmd=get_table : 
tbl=hive.airline_ontime.flights_raw
{code}

The minimum fix needed is to move the Thread.sleep() outside of the try/catch 
block, so that the sleep is triggered even if the attempt throws an exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20452) DPP: Simple n-tuple expressions support

[Gopal V (JIRA)]

Gopal V created HIVE-20452:
--

 Summary: DPP: Simple n-tuple expressions support
 Key: HIVE-20452
 URL: https://issues.apache.org/jira/browse/HIVE-20452
 Project: Hive
  Issue Type: Improvement
  Components: Tez
Affects Versions: 4.0.0
Reporter: Gopal V


{code}
create table sales (s_qty bigint, s_item_sk bigint) partitioned by(s_region 
string, s_date date);

insert into sales partition(s_region='USA', s_date='2018-01-01') values (1,1), 
(2,2);
insert into sales partition(s_region='EU', s_date='2018-02-01') values (1,1), 
(2,2);
{code}

{code}

select count(*) from sales s, agg_sales a where concat(a.s_region, '^', 
a.s_date) = concat(s.s_region, '^', s.s_date) and s.sold_qty > a.median_s_qty 
and a.s_item_sk = s.s_item_sk;

select count(*) from sales s, agg_sales a where (a.s_region, a.s_date) = 
(s.s_region, s.s_date) and s.sold_qty > a.median_s_qty and a.s_item_sk = 
s.s_item_sk; 

select count(*) from sales s, agg_sales a where a.s_region = s.s_region and  
a.s_date = s.s_date and s.sold_qty > a.median_s_qty and a.s_item_sk = 
s.s_item_sk;

{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20419) Vectorization: Speed up VectorizationDispatcher.validateInputFormatAndSchemaEvolution() for ACIDv2

[Gopal V (JIRA)]

Gopal V created HIVE-20419:
--

 Summary: Vectorization: Speed up 
VectorizationDispatcher.validateInputFormatAndSchemaEvolution() for ACIDv2
 Key: HIVE-20419
 URL: https://issues.apache.org/jira/browse/HIVE-20419
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


With ACID table, the format and schema layouts are much more strictly 
controlled - the table cannot be made of partial ORC and partial RCFile.

This assumption can remove this loop and the slow check for schema between each 
partition before vectorizing the operators - the worst-case performance is the 
common & correct case, where all of them match.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20416) LLAP: Fail to package LLAP if the Xmx computed is 0

[Gopal V (JIRA)]

Gopal V created HIVE-20416:
--

 Summary: LLAP: Fail to package LLAP if the Xmx computed is 0
 Key: HIVE-20416
 URL: https://issues.apache.org/jira/browse/HIVE-20416
 Project: Hive
  Issue Type: Bug
  Components: llap
Reporter: Gopal V


{code}
  if (options.getXmx() != -1) {
// Needs more explanation here
// Xmx is not the max heap value in JDK8. You need to subtract 50% of 
the survivor fraction
// from this, to get actual usable memory before it goes into GC
xmx = options.getXmx();
long xmxMb = (xmx / (1024L * 1024L));
conf.setLong(ConfVars.LLAP_DAEMON_MEMORY_PER_INSTANCE_MB.varname, 
xmxMb);

propsDirectOptions.setProperty(ConfVars.LLAP_DAEMON_MEMORY_PER_INSTANCE_MB.varname,
String.valueOf(xmxMb));
{code}

needs a check for the value being so low that start-up (or all queries) will 
fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20385) Date: date + int fails to add days

[Gopal V (JIRA)]

Gopal V created HIVE-20385:
--

 Summary: Date: date + int fails to add days
 Key: HIVE-20385
 URL: https://issues.apache.org/jira/browse/HIVE-20385
 Project: Hive
  Issue Type: Bug
  Components: Types
Affects Versions: 4.0.0
Reporter: Gopal V


{code}
select current_date + 5;

FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments '5': No 
matching method for class 
org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPDTIPlus with (date, int)
{code}

This works in Postgres 9.6 - http://sqlfiddle.com/#!17/9eecb/19253/0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20381) Vectorization: Reduce dedup of GroupBy + PTF turns off vectorization

[Gopal V (JIRA)]

Gopal V created HIVE-20381:
--

 Summary: Vectorization: Reduce dedup of GroupBy + PTF turns off 
vectorization
 Key: HIVE-20381
 URL: https://issues.apache.org/jira/browse/HIVE-20381
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Affects Versions: 3.1.0, 4.0.0
Reporter: Gopal V


One of the PTF Reducers in Query51 is not vectorized because there's a reduce 
deduplication which combines a group-by and a windowing shuffle.

{code}
| Reducer 8  |
| Execution mode: llap   |
| Reduce Vectorization:  |
| enabled: true  |
| enableConditionsMet: hive.vectorized.execution.reduce.enabled 
IS true, hive.execution.engine tez IN [tez, spark] IS true |
| notVectorizedReason: PTF operator: Only PTF directly under 
reduce-shuffle is supported |
| vectorized: false   
{code}

It vectorizes all PTF vertices (after HIVE-20367), with {{ set 
hive.optimize.reducededuplication=false;}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20362) PTF: Invert sorting for CURRENT_ROW to UNBOUNDED FOLLOWING

[Gopal V (JIRA)]

Gopal V created HIVE-20362:
--

 Summary: PTF: Invert sorting for CURRENT_ROW to UNBOUNDED 
FOLLOWING 
 Key: HIVE-20362
 URL: https://issues.apache.org/jira/browse/HIVE-20362
 Project: Hive
  Issue Type: Bug
  Components: CBO, PTF-Windowing
Reporter: Gopal V


{code}
COALESCE(current_page, FIRST_VALUE(current_page,TRUE) OVER ( PARTITION BY 
user_id ORDER BY timestamp ROWS BETWEEN CURRENT ROW and UNBOUNDED FOLLOWING))
{code}

This is a spilling PTF because it needs an unseen row to generate the current 
row when executing this.

This can be inverted, so that the shuffle sorts it the other way, so the result 
can be generated from a previously seen row.

{code}
COALESCE(current_page, LAST_VALUE(current_page,TRUE) OVER ( PARTITION BY 
user_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING and CURRENT ROW))
{code}

This query pattern is bad in general, but seems to exist as a workaround for - 
HIVE-18145



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68281: HIVE-20354

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68281/#review207047
---




ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java
Lines 1000 (patched)
<https://reviews.apache.org/r/68281/#comment290233>

why not save it directly into setHintList()?


- Gopal V


On Aug. 9, 2018, 5:44 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68281/
> ---
> 
> (Updated Aug. 9, 2018, 5:44 p.m.)
> 
> 
> Review request for hive, Eugene Koifman and Jason Dere.
> 
> 
> Bugs: HIVE-20354
> https://issues.apache.org/jira/browse/HIVE-20354
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Semijoin hints dont work with merge statements.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f4d12ae564 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 463880587e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java 
> 8df290435d 
>   ql/src/test/queries/clientpositive/semijoin_hint.q de176affd3 
>   ql/src/test/results/clientpositive/llap/semijoin_hint.q.out 679916de07 
> 
> 
> Diff: https://reviews.apache.org/r/68281/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

[jira] [Created] (HIVE-20338) LLAP: Force synthetic file-id for filesystems which have HDFS protocol impls with POSIX mutation semantics

[Gopal V (JIRA)]

Gopal V created HIVE-20338:
--

 Summary: LLAP: Force synthetic file-id for filesystems which have 
HDFS protocol impls with POSIX mutation semantics
 Key: HIVE-20338
 URL: https://issues.apache.org/jira/browse/HIVE-20338
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


HDFS protocol does not guarantee immutability of files - the synthetic file-id 
includes the mtime of the file as well, which is a fail-safe for such 
filesystems. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20333) CBO: Join removal based on PK-FK declared constraints

[Gopal V (JIRA)]

Gopal V created HIVE-20333:
--

 Summary: CBO: Join removal based on PK-FK declared constraints
 Key: HIVE-20333
 URL: https://issues.apache.org/jira/browse/HIVE-20333
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


A query of the following shape can have its customer join removed entirely on 
the basis of the key containment between customer & store_sales.

{code}
select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales 
,customer where ss_customer_sk = c_customer_sk group by c_customer_sk;
{code}

This query after join removal can be encoded in as 

{code}
select ss_customer_sk as c_customer_sk,sum(ss_quantity*ss_sales_price) ssales 
from store_sales where ss_customer_sk is not null group by ss_customer_sk;
{code}

The rewrite is not applied today and the current PK-FK relationship does not 
allow for a nullable relationship (i.e a declared Foreign Key can't be NULL).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine

[Gopal V (JIRA)]

Gopal V created HIVE-20321:
--

 Summary: Vectorization: Cut down memory size of 1 col 
VectorHashKeyWrapper to <1 CacheLine
 Key: HIVE-20321
 URL: https://issues.apache.org/jira/browse/HIVE-20321
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper is 
bigger than the low Xmx JVMs.

{code}
* 64-bit VM: **
org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals:
 OFFSET  SIZE   
  TYPE DESCRIPTION  VALUE
  016   
   (object header)  N/A
 16 4   
   int VectorHashKeyWrapper.hashcodeN/A
 20 4   
   (alignment/padding gap) 
 24 8   
long[] VectorHashKeyWrapper.longValues  N/A
 32 8 
double[] VectorHashKeyWrapper.doubleValuesN/A
 40 8 
byte[][] VectorHashKeyWrapper.byteValues  N/A
 48 8   
 int[] VectorHashKeyWrapper.byteStarts  N/A
 56 8   
 int[] VectorHashKeyWrapper.byteLengths N/A
 64 8   
org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] 
VectorHashKeyWrapper.decimalValues   N/A
 72 8 
java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A
 80 8 
org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] 
VectorHashKeyWrapper.intervalDayTimeValues   N/A
 88 8
boolean[] VectorHashKeyWrapper.isNull  N/A
 96 8   
org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext 
VectorHashKeyWrapper.hashCtx N/A
Instance size: 104 bytes
Space losses: 4 bytes internal + 0 bytes external = 4 bytes total
{code}

Pulling this up to a parent class allows for this to be cut down to 32 bytes 
for the single column case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20277) Vectorization: Case expressions that return NULL in FILTER

[Gopal V (JIRA)]

Gopal V created HIVE-20277:
--

 Summary: Vectorization: Case expressions that return NULL in FILTER
 Key: HIVE-20277
 URL: https://issues.apache.org/jira/browse/HIVE-20277
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


In cases like Query89, the vertex with the filter is not vectorized.

{code}
   Filter Operator
  predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN 
(((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (null) END (type: 
boolean)
{code}

{code}
Reducer 3 
Execution mode: llap
Reduce Vectorization:
enabled: true
enableConditionsMet: hive.vectorized.execution.reduce.enabled 
IS true, hive.execution.engine tez IN [tez, spark] IS true
notVectorizedReason: FILTER operator: Unexpected hive type name 
void
vectorized: false
{code}

The query specifically has 

{code}
where case when (avg_monthly_sales <> 0) then (abs(sum_sales - 
avg_monthly_sales) / avg_monthly_sales) else null end > 0.1
{code}

while rewriting it to 

{code}
where case when (avg_monthly_sales <> 0) then (abs(sum_sales - 
avg_monthly_sales) / avg_monthly_sales) > 0.1 else false end
{code}

does vectorize into 

{code}
Filter Operator
  Filter Vectorization:
  className: VectorFilterOperator
  native: true
  predicateExpression: SelectColumnIsTrue(col 
12:boolean)(children: VectorUDFAdaptor(CASE WHEN ((avg_window_0 <> 0.0D)) THEN 
(((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false) 
END)(children: DoubleColNotEqualDoubleScalar(col 7:double, val 0.0) -> 
8:boolean, DoubleColGreaterDoubleScalar(col 9:double, val 0.1)(children: 
DoubleColDivideDoubleColumn(col 10:double, col 7:double)(children: 
FuncAbsDoubleToDouble(col 9:double)(children: DoubleColSubtractDoubleColumn(col 
6:double, col 7:double) -> 9:double) -> 10:double) -> 9:double) -> 11:boolean) 
-> 12:boolean)
  predicate: CASE WHEN ((avg_window_0 <> 0.0D)) THEN 
(((abs((_col6 - avg_window_0)) / avg_window_0) > 0.1D)) ELSE (false) END (type: 
boolean)
  Statistics: Num rows: 11 Data size: 5291 Basic stats: 
COMPLETE Column stats: COMPLETE
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68115: HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68115/#review206627
---




serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
Lines 76 (patched)
<https://reviews.apache.org/r/68115/#comment289673>

The type -> ObjectInspector is the only one which I think looks like it is 
there for "intern-ing" of object inpsectors (So that object == can be used 
instead of a deep equals check).

Of course, I'm not sure that's a "real" problem unless a query takes > 5 
minutes to compile.

If that doesn't show up in the leaks, it might better to leave that one 
alone.


- Gopal V


On July 31, 2018, 12:08 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68115/
> ---
> 
> (Updated July 31, 2018, 12:08 a.m.)
> 
> 
> Review request for hive, Gopal V and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-20274
> https://issues.apache.org/jira/browse/HIVE-20274
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20274: HiveServer2 ObjectInspectorFactory leaks for Struct and List 
> object inspectors
> 
> 
> Diffs
> -
> 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
>  b45a9dee72f5fb3a139ee205a05e23c068b1fa0c 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
>  3a88a400250afa0e29001ec118e006f937678089 
>   
> serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
>  cb1440e98b8fad399c4c02a7440d2a30483455f4 
> 
> 
> Diff: https://reviews.apache.org/r/68115/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

[jira] [Created] (HIVE-20274) HiveServer2 ObjectInspectorFactory leaks for Struct and List object inspectors

[Gopal V (JIRA)]

Gopal V created HIVE-20274:
--

 Summary: HiveServer2 ObjectInspectorFactory leaks for Struct and 
List object inspectors
 Key: HIVE-20274
 URL: https://issues.apache.org/jira/browse/HIVE-20274
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.0.0, 4.0.0
Reporter: Gopal V


Fix in HIVE-19860 needs to be applied to 

{code}
  static ConcurrentHashMap
  cachedStandardListObjectInspector = new 
ConcurrentHashMap();
...
  static ConcurrentHashMap>, StandardStructObjectInspector> 
cachedStandardStructObjectInspector =
  new ConcurrentHashMap>, 
StandardStructObjectInspector>();
...
  static ConcurrentHashMap, ColumnarStructObjectInspector> 
cachedColumnarStructObjectInspector =
  new ConcurrentHashMap, ColumnarStructObjectInspector>();
{code}

And possibly for 

{code}
  static ConcurrentHashMap objectInspectorCache = new 
ConcurrentHashMap();
...
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/#review206233
---


Ship it!





common/src/java/org/apache/hive/http/ProfileServlet.java
Lines 211 (patched)
<https://reviews.apache.org/r/67966/#comment289122>

final boolean variables for "thread", "simple" and "reverse"


- Gopal V


On July 19, 2018, 1:08 a.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67966/
> ---
> 
> (Updated July 19, 2018, 1:08 a.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-20202
> https://issues.apache.org/jira/browse/HIVE-20202
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20202: Add profiler endpoint to httpserver
> 
> 
> Diffs
> -
> 
>   bin/hive 1ade51e 
>   common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
>   common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67966/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/#review206218
---




common/src/java/org/apache/hive/http/HttpServer.java
Lines 561 (patched)
<https://reviews.apache.org/r/67966/#comment289108>

Enable this only if the "async.profiler.home" property is setup - so that 
this is not enabled accidentally for anyone.


- Gopal V


On July 18, 2018, 6:42 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67966/
> ---
> 
> (Updated July 18, 2018, 6:42 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-20202
> https://issues.apache.org/jira/browse/HIVE-20202
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20202: Add profiler endpoint to httpserver
> 
> 
> Diffs
> -
> 
>   bin/hive 1ade51e 
>   common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
>   common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67966/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

Re: Review Request 67966: HIVE-20202: Add profiler endpoint to httpserver

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67966/#review206217
---




common/src/java/org/apache/hive/http/ProfileServlet.java
Lines 172 (patched)
<https://reviews.apache.org/r/67966/#comment289107>

Input validation (most of these are typed, as far as I know)



common/src/java/org/apache/hive/http/ProfileServlet.java
Lines 184 (patched)
<https://reviews.apache.org/r/67966/#comment289106>

Change to a tryLock + timeout - fail with an error if the GET request can't 
get a lock after a while.


- Gopal V


On July 18, 2018, 6:42 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67966/
> ---
> 
> (Updated July 18, 2018, 6:42 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-20202
> https://issues.apache.org/jira/browse/HIVE-20202
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-20202: Add profiler endpoint to httpserver
> 
> 
> Diffs
> -
> 
>   bin/hive 1ade51e 
>   common/src/java/org/apache/hive/common/util/ProcessUtils.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java 3cb7a33 
>   common/src/java/org/apache/hive/http/ProfileServlet.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67966/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>

[jira] [Created] (HIVE-20177) Vectorization: Reduce KeyWrapper allocation in GroupBy Streaming mode

[Gopal V (JIRA)]

Gopal V created HIVE-20177:
--

 Summary: Vectorization: Reduce KeyWrapper allocation in GroupBy 
Streaming mode
 Key: HIVE-20177
 URL: https://issues.apache.org/jira/browse/HIVE-20177
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Reporter: Gopal V


The streaming mode for VectorGroupBy allocates a large number of arrays due to 
VectorKeyHashWrapper::duplicateTo()

Since the vectors can't be mutated in-place while a single batch is being 
processed, this operation can be cut by 1000x by allocating a streaming key at 
the end of the loop, instead of reallocating within the loop.

{code}
  for(int i = 0; i < batch.size; ++i) {
if (!batchKeys[i].equals(streamingKey)) {
  // We've encountered a new key, must save current one
  // We can't forward yet, the aggregators have not been evaluated
  rowsToFlush[flushMark] = currentStreamingAggregators;
  if (keysToFlush[flushMark] == null) {
keysToFlush[flushMark] = (VectorHashKeyWrapper) 
streamingKey.copyKey();
  } else {
streamingKey.duplicateTo(keysToFlush[flushMark]);
  }

  currentStreamingAggregators = 
streamAggregationBufferRowPool.getFromPool();
  batchKeys[i].duplicateTo(streamingKey);
  ++flushMark;
}
{code}

The duplicateTo can be pushed out of the loop since there only one to truly 
keep a copy of is the last unique key in the VRB.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20176) ResultsCache: Time based invalidation is not required for transactional cache entries

[Gopal V (JIRA)]

Gopal V created HIVE-20176:
--

 Summary: ResultsCache: Time based invalidation is not required for 
transactional cache entries
 Key: HIVE-20176
 URL: https://issues.apache.org/jira/browse/HIVE-20176
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0
Reporter: Gopal V


If a cache entry has explicit transactional validity, it does not need to be 
invalidated on the timer - the access by a later write-id state will ignore it 
without having to invalidate it on the timer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20139) LLAP: execution.mode='only' ignores custom UDTFs

[Gopal V (JIRA)]

Gopal V created HIVE-20139:
--

 Summary: LLAP: execution.mode='only' ignores custom UDTFs
 Key: HIVE-20139
 URL: https://issues.apache.org/jira/browse/HIVE-20139
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


The LLAP UDF/UDAF check needs to be extended to UDTFs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20138) Sessions: Closing Tez sessions if opened outside of default sessions throws NPE

[Gopal V (JIRA)]

Gopal V created HIVE-20138:
--

 Summary: Sessions: Closing Tez sessions if opened outside of 
default sessions throws NPE
 Key: HIVE-20138
 URL: https://issues.apache.org/jira/browse/HIVE-20138
 Project: Hive
  Issue Type: Bug
  Components: Tez
Reporter: Gopal V


On a busy cluster, if a session is currently in ACCEPTED state (i.e YARN hasn't 
started it yet), running a query will NPE.

{code}
java.lang.NullPointerException: null
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.getSession(TezSessionState.java:711)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.close(TezSessionState.java:646)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:353)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:467)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.tez.WorkloadManagerFederation.getUnmanagedSession(WorkloadManagerFederation.java:66)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.exec.tez.WorkloadManagerFederation.getSession(WorkloadManagerFederation.java:38)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:184) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle

[Gopal V (JIRA)]

Gopal V created HIVE-20113:
--

 Summary: Shuffle avoidance: Disable 1-1 edges for sorted shuffle 
 Key: HIVE-20113
 URL: https://issues.apache.org/jira/browse/HIVE-20113
 Project: Hive
  Issue Type: Bug
  Components: Tez
Reporter: Gopal V


The sorted shuffle avoidance can have some issues when the shuffle data gets 
broken up into multiple chunks on disk.

The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to 
have a final merge at all, it should open a single compressed file and write a 
single index entry.

Until the shuffle issue is resolved & a lot more testing, it is prudent to 
disable the optimization for sorted shuffle edges and stop rewriting the 
RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20103) WM: Only Aggregate DAG counters if at least one is used

[Gopal V (JIRA)]

Gopal V created HIVE-20103:
--

 Summary: WM: Only Aggregate DAG counters if at least one is used
 Key: HIVE-20103
 URL: https://issues.apache.org/jira/browse/HIVE-20103
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 4.0.0
Reporter: Gopal V


{code}
status = dagClient.getDAGStatus(EnumSet.of(StatusGetOpts.GET_COUNTERS), 
checkInterval);
TezCounters dagCounters = status.getDAGCounters();
...
if (dagCounters != null && wmContext != null) {
  Set desiredCounters = wmContext.getSubscribedCounters();
  if (desiredCounters != null && !desiredCounters.isEmpty()) {
Map currentCounters = getCounterValues(dagCounters, 
vertexNames, vertexProgressMap,
  desiredCounters, done);
{code}

Skip collecting DAG counters unless there at least one desired counter in 
wmContext.

The AM has a hard-lock around the counters, so the current jstacks are full of 

{code}
   java.lang.Thread.State: RUNNABLE
at java.lang.String.intern(Native Method)
at 
org.apache.hadoop.util.StringInterner.weakIntern(StringInterner.java:71)
at 
org.apache.tez.common.counters.GenericCounter.(GenericCounter.java:50)
at 
org.apache.tez.common.counters.TezCounters$GenericGroup.newCounter(TezCounters.java:65)
at 
org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92)
at 
org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:104)
- locked <0x7efb3ac7af38> (a 
org.apache.tez.common.counters.TezCounters$GenericGroup)
at 
org.apache.tez.common.counters.AbstractCounterGroup.aggrAllCounters(AbstractCounterGroup.java:204)
at 
org.apache.tez.common.counters.AbstractCounters.aggrAllCounters(AbstractCounters.java:372)
- eliminated <0x7efb3ac64ee8> (a 
org.apache.tez.common.counters.TezCounters)
at 
org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:357)
- locked <0x7efb3ac64ee8> (a 
org.apache.tez.common.counters.TezCounters)
at 
org.apache.tez.dag.app.dag.impl.TaskImpl.getCounters(TaskImpl.java:462)
at 
org.apache.tez.dag.app.dag.impl.VertexImpl.aggrTaskCounters(VertexImpl.java:1342)
at 
org.apache.tez.dag.app.dag.impl.VertexImpl.getAllCounters(VertexImpl.java:1202)
at 
org.apache.tez.dag.app.dag.impl.DAGImpl.aggrTaskCounters(DAGImpl.java:755)
at 
org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:704)
at 
org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:901)
at 
org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:940)
at 
org.apache.tez.dag.api.client.DAGClientHandler.getDAGStatus(DAGClientHandler.java:73)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20101) BloomKFilter: Avoid using the local byte[] arrays entirely

[Gopal V (JIRA)]

Gopal V created HIVE-20101:
--

 Summary: BloomKFilter: Avoid using the local byte[] arrays entirely
 Key: HIVE-20101
 URL: https://issues.apache.org/jira/browse/HIVE-20101
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 4.0.0
Reporter: Gopal V


HIVE-18866 introduced a fast-path for integer -> murmur hash, but the change 
hasn't been applied to BloomKFilter for integers.

{code}
public class BloomKFilter {
  private final byte[] BYTE_ARRAY_4 = new byte[4];
  private final byte[] BYTE_ARRAY_8 = new byte[8];
{code}

Remove these objects and use the fast-path.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67800: HIVE-20039

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/#review205687
---


Ship it!




Ship It!

- Gopal V


On July 3, 2018, 8:20 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67800/
> ---
> 
> (Updated July 3, 2018, 8:20 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-20039
> https://issues.apache.org/jira/browse/HIVE-20039
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Bucket pruning: Left Outer Join on bucketed table gives wrong result.
> The context was reused by all the predicates. Instead use TS op directly.
> 
> 
> Diffs
> -
> 
>   
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/00_0
>  PRE-CREATION 
>   
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/00_0
>  PRE-CREATION 
>   data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/00_0 
> PRE-CREATION 
>   data/files/bucket_pruning/l3_monthly_dw_dimplan/56_0 PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties d02c0fe8ba 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
>  2debacacb5 
>   ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67800/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 67800: HIVE-20039

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/#review205686
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
Lines 46 (patched)
<https://reviews.apache.org/r/67800/#comment288585>

minor nit


- Gopal V


On July 3, 2018, 8:20 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67800/
> ---
> 
> (Updated July 3, 2018, 8:20 p.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-20039
> https://issues.apache.org/jira/browse/HIVE-20039
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Bucket pruning: Left Outer Join on bucketed table gives wrong result.
> The context was reused by all the predicates. Instead use TS op directly.
> 
> 
> Diffs
> -
> 
>   
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/00_0
>  PRE-CREATION 
>   
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/00_0
>  PRE-CREATION 
>   data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/00_0 
> PRE-CREATION 
>   data/files/bucket_pruning/l3_monthly_dw_dimplan/56_0 PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties d02c0fe8ba 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
>  2debacacb5 
>   ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67800/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

Re: Review Request 67800: HIVE-20039

[Gopal V]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67800/#review205649
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
Lines 90 (patched)
<https://reviews.apache.org/r/67800/#comment288536>

This needs the pruned partition list loop (to make sure all partitions have 
the same bucket-count).

This is probably something we can prevent from happening in Bucketing 
version = 2 so that tables with new bucketing version will never allow an 

ALTER TABLE clustered by (id) buckets 

from changing a partition's bucket count.


- Gopal V


On July 3, 2018, 12:23 a.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67800/
> ---
> 
> (Updated July 3, 2018, 12:23 a.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-20039
> https://issues.apache.org/jira/browse/HIVE-20039
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Bucket pruning: Left Outer Join on bucketed table gives wrong result.
> The context was reused by all the predicates. Instead use TS op directly.
> 
> 
> Diffs
> -
> 
>   
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_datajoin_1_s2_2018022300104_1/00_0
>  PRE-CREATION 
>   
> data/files/bucket_pruning/l3_clarity__l3_monthly_dw_factplan_dw_stg_2018022300104_1/00_0
>  PRE-CREATION 
>   data/files/bucket_pruning/l3_clarity__l3_snap_number_2018022300104/00_0 
> PRE-CREATION 
>   data/files/bucket_pruning/l3_monthly_dw_dimplan/56_0 PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
>  2debacacb5 
>   ql/src/test/queries/clientpositive/tez_fixed_bucket_pruning.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/tez_fixed_bucket_pruning.q.out 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67800/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>

[jira] [Created] (HIVE-20043) HiveServer2: SessionState has a static sync block around an AtomicBoolean

[Gopal V (JIRA)]

Gopal V created HIVE-20043:
--

 Summary: HiveServer2: SessionState has a static sync block around 
an AtomicBoolean
 Key: HIVE-20043
 URL: https://issues.apache.org/jira/browse/HIVE-20043
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: Gopal V


{code}
  private static void start(SessionState startSs, boolean isAsync, LogHelper 
console) {
...
synchronized(SessionState.class) {
  if (!startSs.isStarted.compareAndSet(false, true)) {
return;
  }
}
{code}

startSs.isStarted is an AtomicBoolean, which makes it hard to know why this 
code is locked with a static lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20042) HiveServer2: All operations lock on a Single HiveConf object

[Gopal V (JIRA)]

Gopal V created HIVE-20042:
--

 Summary: HiveServer2: All operations lock on a Single HiveConf 
object 
 Key: HIVE-20042
 URL: https://issues.apache.org/jira/browse/HIVE-20042
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: Gopal V


With the 1000 user test, the session start/tear-down runs only at 100% CPU, 
which is due to all threads locking on the same HiveConf object.

OpenSession locks on 0x0005c091a3a0

{code}
"HiveServer2-HttpHandler-Pool: Thread-65084" #65084 prio=5 os_prio=0 
tid=0x103bb000 nid=0x4a09 waiting for monitor entry [0x7fc1b0987000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.conf.Configuration.getOverlay(Configuration.java:1418)
- waiting to lock <0x0005c091a3a0> (a 
org.apache.hadoop.hive.conf.HiveConf)
at 
org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:711)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1437)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4996)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5069)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.getUserName(ThriftCLIService.java:424)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:467)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:325)
{code}

GetOperationStatus locks on the same 

{code}
"HiveServer2-HttpHandler-Pool: Thread-65082" #65082 prio=5 os_prio=0 
tid=0x7fc2656be000 nid=0x4a06 waiting for monitor entry [0x7fc3159db000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.conf.Configuration.getOverlay(Configuration.java:1418)
- waiting to lock <0x0005c091a3a0> (a 
org.apache.hadoop.hive.conf.HiveConf)
at 
org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:711)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1437)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4996)
at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5069)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.GetOperationStatus(ThriftCLIService.java:709)
{code}

Session clean up locks on the same

{code}
"8fd1db09-9f96-49dc-becf-5702826bd4f5 HiveServer2-HttpHandler-Pool: 
Thread-64981" #64981 prio=5 os_prio=0 tid=0x1d1ab000 nid=0x23d5 waiting 
for monitor entry [0x7fc1b65e3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.conf.Configuration.getOverlay(Configuration.java:1418)
- waiting to lock <0x0005c091a3a0> (a 
org.apache.hadoop.hive.conf.HiveConf)
at 
org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:711)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1177)
at 
org.apache.hadoop.conf.Configuration.getTrimmedStringCollection(Configuration.java:2122)
at 
org.apache.hadoop.hdfs.DFSUtilClient.getNameServiceIds(DFSUtilClient.java:197)
at org.apache.hadoop.hdfs.HAUtilClient.isLogicalUri(HAUtilClient.java:53)
...
at 
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:959)
at org.apache.hadoop.hive.ql.Context.clear(Context.java:724)
{code}

Hadoop RPC blocks on the same

{code}
"HiveServer2-HttpHandler-Pool: Thread-59227" #59227 prio=5 os_prio=0 
tid=0x7fc270aeb800 nid=0x129b waiting for monitor entry [0x7fc28b7b5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.conf.Configuration.getOverlay(Configuration.java:1418)
- waiting to lock <0x0005c091a3a0> (a 
org.apache.hadoop.hive.conf.HiveConf)
at 
org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:711)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1177)
at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1234)
at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1459)
at org.apache.hadoop.ipc.Client$Connection.(Client.java:451)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1532)
 ...
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1580)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
at org.apache.hadoop.fs.FileSystem.deleteOnExit(FileSystem.java:1677)
at 
org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:791)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20041) ResultsCache: Improve loggng for concurrent queries

[Gopal V (JIRA)]

Gopal V created HIVE-20041:
--

 Summary: ResultsCache: Improve loggng for concurrent queries
 Key: HIVE-20041
 URL: https://issues.apache.org/jira/browse/HIVE-20041
 Project: Hive
  Issue Type: Improvement
  Components: Diagnosability
Reporter: Gopal V


The logging for QueryResultsCache ends up printing information without context, 
like 

{code}
2018-06-30T17:48:45,502  INFO [HiveServer2-Background-Pool: Thread-166] 
results.QueryResultsCache: Waiting on pending cacheEntry
{code}

{code}
2018-06-30T17:50:17,963  INFO [HiveServer2-Background-Pool: Thread-145] 
ql.Driver: savedToCache: true
{code}

The previous lines for this are in DEBUG level, so the logging ends up being 
useless at INFO level to debug.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20040) JDBC: HTTP listen queue is 50 and SYNs are lost

[Gopal V (JIRA)]

Gopal V created HIVE-20040:
--

 Summary: JDBC: HTTP listen queue is 50 and SYNs are lost
 Key: HIVE-20040
 URL: https://issues.apache.org/jira/browse/HIVE-20040
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


When testing with 5000 concurrent users, the JDBC HTTP port ends up overflowing 
on SYNs when the HS2 gc pauses.

This is because each getQueryProgress request is an independent HTTP request, 
so unlike the BINARY mode, there are more connections being established & 
closed in HTTP mode.

{code}
LISTEN 0  50   *:10004*:* 
{code}

This turns into connection errors when enabling 
{{net.ipv4.tcp_abort_on_overflow=1}}, but the better approach is to enqueue the 
connections until the HS2 is done with its GC pause.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20021) LLAP: Fall back to Synthetic File-ids when getting a HdfsConstants.GRANDFATHER_INODE_ID

[Gopal V (JIRA)]

Gopal V created HIVE-20021:
--

 Summary: LLAP: Fall back to Synthetic File-ids when getting a 
HdfsConstants.GRANDFATHER_INODE_ID
 Key: HIVE-20021
 URL: https://issues.apache.org/jira/browse/HIVE-20021
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


HDFS client implementations have multiple server implementations, which do not 
all support the inodes for file locations.

If the client returns a 0 InodeId, fall back to the synthetic ones.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19985) ACID: Skip decoding the ROW__ID sections for read-only queries

[Gopal V (JIRA)]

Gopal V created HIVE-19985:
--

 Summary: ACID: Skip decoding the ROW__ID sections for read-only 
queries 
 Key: HIVE-19985
 URL: https://issues.apache.org/jira/browse/HIVE-19985
 Project: Hive
  Issue Type: Improvement
  Components: Transactions
Reporter: Gopal V


For a base_n file there are no aborted transactions within the file and if 
there are no pending delete deltas, the entire ACID ROW__ID can be skipped for 
all read-only queries (i.e SELECT), though it still needs to be projected out 
for MERGE, UPDATE and DELETE queries.

This patch tries to entirely ignore the ACID ROW__ID fields for all tables 
where there are no possible deletes or aborted transactions for an ACID split.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)