[jira] [Created] (HIVE-17207) Hiveserver2 fails to start when hive.server2.authentication is set to KERBEROS

[Rey Rey Chang (JIRA)]

Rey Rey Chang created HIVE-17207:


 Summary: Hiveserver2 fails to start when 
hive.server2.authentication is set to KERBEROS
 Key: HIVE-17207
 URL: https://issues.apache.org/jira/browse/HIVE-17207
 Project: Hive
  Issue Type: Bug
  Components: Authentication
Affects Versions: 2.2.0
Reporter: Rey Rey Chang


The logic in getAuthTransFactory() is flawed, as it will always "Unsupported 
authentication type" error when authTypeStr is set to KERBEROS. 

inside HiveAuthFactory class.
  public TTransportFactory getAuthTransFactory()
throws LoginException
  {
TTransportFactory transportFactory;
if (isSASLWithKerberizedHadoop())
{
  try
  {
serverTransportFactory = 
this.saslServer.createSaslServerTransportFactory(
  getSaslProperties());
  }
  catch (TTransportException e)
  {
TSaslServerTransport.Factory serverTransportFactory;
throw new LoginException(e.getMessage());
  }
  TSaslServerTransport.Factory serverTransportFactory;
  if (!this.authTypeStr.equalsIgnoreCase(AuthTypes.KERBEROS.getAuthName())) 
{
if ((this.authTypeStr.equalsIgnoreCase(AuthTypes.NONE.getAuthName())) 
|| 
  (this.authTypeStr.equalsIgnoreCase(AuthTypes.LDAP.getAuthName())) || 
  (this.authTypeStr.equalsIgnoreCase(AuthTypes.PAM.getAuthName())) || 
  (this.authTypeStr.equalsIgnoreCase(AuthTypes.CUSTOM.getAuthName( {
  try
  {
serverTransportFactory.addServerDefinition("PLAIN", 
this.authTypeStr, null, new HashMap(), new 
PlainSaslHelper.PlainServerCallbackHandler(this.authTypeStr));
  }
  catch (AuthenticationException e)
  {
throw new LoginException("Error setting callback handler" + e);
  }
} else {
  throw new LoginException("Unsupported authentication type " + 
this.authTypeStr);
}
  }
  transportFactory = 
this.saslServer.wrapTransportFactory(serverTransportFactory);
}
else
{
  TTransportFactory transportFactory;
  if ((this.authTypeStr.equalsIgnoreCase(AuthTypes.NONE.getAuthName())) || 
(this.authTypeStr.equalsIgnoreCase(AuthTypes.LDAP.getAuthName())) || 
(this.authTypeStr.equalsIgnoreCase(AuthTypes.PAM.getAuthName())) || 
(this.authTypeStr.equalsIgnoreCase(AuthTypes.CUSTOM.getAuthName(
  {
transportFactory = 
PlainSaslHelper.getPlainTransportFactory(this.authTypeStr);
  }
  else
  {
TTransportFactory transportFactory;
if (this.authTypeStr.equalsIgnoreCase(AuthTypes.NOSASL.getAuthName())) {
  transportFactory = new TTransportFactory();
} else {
  throw new LoginException("Unsupported authentication type " + 
this.authTypeStr);
}
  }
}
TTransportFactory transportFactory;
return transportFactory;
  }



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17206) make a version of Compactor specific to unbucketed tables

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-17206:
-

 Summary: make a version of Compactor specific to unbucketed tables
 Key: HIVE-17206
 URL: https://issues.apache.org/jira/browse/HIVE-17206
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Reporter: Eugene Koifman
Assignee: Eugene Koifman


current Compactor will work but is not optimized/flexible enough



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17205) add functional support

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-17205:
-

 Summary: add functional support
 Key: HIVE-17205
 URL: https://issues.apache.org/jira/browse/HIVE-17205
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Reporter: Eugene Koifman
Assignee: Eugene Koifman


make sure unbucketed tables can be marked transactional=true
make insert/update/delete/compaction work



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17204) support un-bucketed tables in acid

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-17204:
-

 Summary: support un-bucketed tables in acid
 Key: HIVE-17204
 URL: https://issues.apache.org/jira/browse/HIVE-17204
 Project: Hive
  Issue Type: New Feature
  Components: Transactions
Reporter: Eugene Koifman
Assignee: Eugene Koifman






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 59402: HIVE-16718: Provide a way to pass in user supplied maven build and test arguments to Ptest

[Sahil Takiar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59402/#review181710
---



Overall it looks good. What about other Maven options though? Would it be 
difficult to update this patch so that any option for `mvn` can be passed in? 
Not just options specified by `-D`?

- Sahil Takiar


On May 19, 2017, 3:10 p.m., Barna Zsombor Klara wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59402/
> ---
> 
> (Updated May 19, 2017, 3:10 p.m.)
> 
> 
> Review request for hive, Sergio Pena and Sahil Takiar.
> 
> 
> Bugs: HIVE-16718
> https://issues.apache.org/jira/browse/HIVE-16718
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16718: Provide a way to pass in user supplied maven build and test 
> arguments to Ptest
> 
> 
> Diffs
> -
> 
>   testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java 
> 1cdfdb309acd8282e593abd7ed10c87721926c60 
>   
> testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/TestConfiguration.java
>  e584f9c105fa134e3e267d6c6817d441b4c6b249 
> 
> 
> Diff: https://reviews.apache.org/r/59402/diff/1/
> 
> 
> Testing
> ---
> 
> Tested manually that I can pass in 0, 1 or 2 arguments to both mvnBuildArgs 
> and mvnTestArgs in PTest.
> Arguments will be key value pairs separated by '='. The '-D' will be added 
> automatically.
> 
> 
> Thanks,
> 
> Barna Zsombor Klara
> 
>

[jira] [Created] (HIVE-17203) Add InterfaceAudience and InterfaceStability annotations for HCat APIs

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-17203:
---

 Summary: Add InterfaceAudience and InterfaceStability annotations 
for HCat APIs
 Key: HIVE-17203
 URL: https://issues.apache.org/jira/browse/HIVE-17203
 Project: Hive
  Issue Type: Sub-task
  Components: HCatalog
Affects Versions: 3.0.0
Reporter: Sahil Takiar
Assignee: Sahil Takiar






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17202) Add InterfaceAudience and InterfaceStability annotations for HMS Listener APIs

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-17202:
---

 Summary: Add InterfaceAudience and InterfaceStability annotations 
for HMS Listener APIs
 Key: HIVE-17202
 URL: https://issues.apache.org/jira/browse/HIVE-17202
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Sahil Takiar
Assignee: Sahil Takiar






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17201) (Temporarily) Disable failing tests in TestHCatClient

[Mithun Radhakrishnan (JIRA)]

Mithun Radhakrishnan created HIVE-17201:
---

 Summary: (Temporarily) Disable failing tests in TestHCatClient
 Key: HIVE-17201
 URL: https://issues.apache.org/jira/browse/HIVE-17201
 Project: Hive
  Issue Type: Bug
  Components: HCatalog, Tests
Affects Versions: 3.0.0
Reporter: Mithun Radhakrishnan
Assignee: Mithun Radhakrishnan


This is with regard to the recent test-failures in {{TestHCatClient}}. 

While [~sbeeram] and I joust over the best way to rephrase the failing tests 
(in HIVE-16908), perhaps it's best that we temporarily disable the following 
failing tests:
{noformat}
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=177)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=177)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

InterfaceAudience + InterfaceStability Annotations and jacc

[Sahil Takiar]

Hello Everyone,

Wanted to bring up some work I am doing in Hive to improve how we detect
backwards incompatible changes to core APIs (HIVE-17129
 and HIVE-17130
).

Hive contains InterfaceAudience and InterfaceStability annotations to mark
specific Java classes as Public APIs. The goal of these annotations to help
mark what parts of Hive can be considered part of the Hive API.

The annotations were introduced a while ago, but have only been applied to
a few Java classes. I've filed HIVE-17129
 to update the usage of
these annotations so they apply to more parts of Hive, such as the SerDe
and UDF APIs. If anyone has any comments or suggestions on what should and
shouldn't be considered an API, feel free to comment on the JIRA.

One major advantage of using these annotations is that it allows us to use
tools such as the Java API Compliance Checker
. This tool checks for
backwards incompatible changes to Java classes tagged with a specific
annotation. It is used by a number of other Apache projects such as Hadoop,
HBase, Kudu, and Spark (Spark uses a Scala equivalent). The tool provides
us with an automated way of detecting backwards incompatible changes to a
given API. We could setup a dedicated Jenkins job to run the tool, or even
build it into the pre-commit checks. I've filed HIVE-17130
 to track this work.

Wanted to surface this on the dev list in case anyone has comments on what
should be considered a Public API and what shouldn't be.

Thanks!

--Sahil Takiar

[jira] [Created] (HIVE-17200) TestMiniSparkOnYarnCliDriver: [spark_vectorized_dynamic_partition_pruning]

[Janaki Lahorani (JIRA)]

Janaki Lahorani created HIVE-17200:
--

 Summary: TestMiniSparkOnYarnCliDriver: 
[spark_vectorized_dynamic_partition_pruning]
 Key: HIVE-17200
 URL: https://issues.apache.org/jira/browse/HIVE-17200
 Project: Hive
  Issue Type: Sub-task
Reporter: Janaki Lahorani


Error:
Client execution failed with error code = 3 running "
select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart 
group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'" 
fname=spark_vectorized_dynamic_partition_pruning.q 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17199) TestHiveMetaStoreStatsMerge: testStatsMerge

[Janaki Lahorani (JIRA)]

Janaki Lahorani created HIVE-17199:
--

 Summary: TestHiveMetaStoreStatsMerge: testStatsMerge
 Key: HIVE-17199
 URL: https://issues.apache.org/jira/browse/HIVE-17199
 Project: Hive
  Issue Type: Sub-task
Reporter: Janaki Lahorani


Stack:
org.apache.thrift.transport.TTransportException
at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at 
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at 
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_aggr_stats_for(ThriftHiveMetastore.java:3724)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_aggr_stats_for(ThriftHiveMetastore.java:3711)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:1704)
at 
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge(TestHiveMetaStoreStatsMerge.java:187)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]

[Janaki Lahorani (JIRA)]

Janaki Lahorani created HIVE-17198:
--

 Summary: Flaky test: TestBeeLineDriver [smb_mapjoin_7]
 Key: HIVE-17198
 URL: https://issues.apache.org/jira/browse/HIVE-17198
 Project: Hive
  Issue Type: Sub-task
Reporter: Janaki Lahorani


Error:
Exception running or analyzing the results of the query file: 
org.apache.hive.beeline.QFile@4f7b68ad



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17197) Flaky test: TestMiniSparkOnYarnCliDriver

[Janaki Lahorani (JIRA)]

Janaki Lahorani created HIVE-17197:
--

 Summary: Flaky test: TestMiniSparkOnYarnCliDriver
 Key: HIVE-17197
 URL: https://issues.apache.org/jira/browse/HIVE-17197
 Project: Hive
  Issue Type: Sub-task
Reporter: Janaki Lahorani


Error:
Failed during createSources processLine with code=3




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] hive pull request #199: HIVE-16750: Support change management for rename tab...

[sankarh]

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/199


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hive pull request #200: HIVE-16901: Distcp optimization - One distcp per Rep...

[sankarh]

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/200


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] hive pull request #212: HIVE-17195: Long chain of tasks created by REPL LOAD...

[sankarh]

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/212

HIVE-17195: Long chain of tasks created by REPL LOAD shouldn't cause stack 
corruption.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-17195

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/212.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #212


commit 64cc709e8591eaee1b22aaf0bb6144c33259e058
Author: Sankar Hariappan 
Date:   2017-07-28T10:53:43Z

HIVE-17195: Long chain of tasks created by REPL LOAD shouldn't cause stack 
corruption.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HIVE-17196) CM: ReplCopyTask should retain the original file names even if copied from CM path.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-17196:
---

 Summary: CM: ReplCopyTask should retain the original file names 
even if copied from CM path.
 Key: HIVE-17196
 URL: https://issues.apache.org/jira/browse/HIVE-17196
 Project: Hive
  Issue Type: Sub-task
  Components: repl
Affects Versions: 2.1.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Consider the below scenario,
1. Insert into table T1 with value(X).
2. Insert into table T1 with value(X).
3. Truncate the table T1. 
– This step backs up 2 files with same content to cmroot which ends up with one 
file in cmroot as checksum matches.
4. Incremental repl with above 3 operations.
– In this step, both the insert event files will be read from cmroot where copy 
of one leads to overwrite the other one as the file name is same in cm path 
(checksum as file name).
So, this leads to data loss and hence it is necessary to retain the original 
file names even if we copy from cm path.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17195) Long chain of tasks created by REPL LOAD shouldn't cause stack corruption.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-17195:
---

 Summary: Long chain of tasks created by REPL LOAD shouldn't cause 
stack corruption.
 Key: HIVE-17195
 URL: https://issues.apache.org/jira/browse/HIVE-17195
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2, repl
Affects Versions: 2.1.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Currently, long chain REPL LOAD tasks lead to huge recursive calls when try to 
traverse the DAG.
For example, getMRTasks, getTezTasks, getSparkTasks and iterateTasks methods 
run recursively to traverse the DAG.
Need to modify this traversal logic to reduce stack usage.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Separating out the metastore as its own TLP

[Gopal Vijayaraghavan]

On 7/25/17, 4:45 PM, "cwsteinb...@gmail.com on behalf of Carl Steinbach" 
 wrote:

>"IceWeasel" and "MetaStore" are both examples of English compound words.
>What exactly makes the former any safer than the latter?

Usually descriptive words are considered weaker for trademarks - if the words 
describe what it does, then it might be weaker.

"PainKiller" is a weak one, while "Aspirin" isn't. 

Uniqueness is useful, because an active defense is necessary to retain 
possession of a trademark - as a tautology, the more unique the phrase, the 
fewer occurrences there are to tackle.
 
But, in the case of Aspirin, Bayer did not defend the use lowercase "aspirin" 
and now only has a TM on the upper-case one "Aspirin".

IceWeasel is an infamous precedent of trademark dispute in the open source 
community

"The end of the Iceweasel Age" - https://lwn.net/Articles/676799/
+
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=354622

Cheers,
Gopal

[jira] [Created] (HIVE-17194) JDBC: Implement Gzip servlet filter

[Gopal V (JIRA)]

Gopal V created HIVE-17194:
--

 Summary: JDBC: Implement Gzip servlet filter
 Key: HIVE-17194
 URL: https://issues.apache.org/jira/browse/HIVE-17194
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2, JDBC
Affects Versions: 3.0.0
Reporter: Gopal V


{code}
POST /cliservice HTTP/1.1
Content-Type: application/x-thrift
Accept: application/x-thrift
User-Agent: Java/THttpClient/HC
Authorization: Basic YW5vbnltb3VzOmFub255bW91cw==
Content-Length: 71
Host: localhost:10007
Connection: Keep-Alive
Accept-Encoding: gzip,deflate
X-XSRF-HEADER: true
{code}

The Beeline client clearly sends out HTTP compression headers which are ignored 
by the HTTP service layer in HS2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)