[jira] [Created] (HIVE-21428) field delimiter of serde set for partition is not getting respected when vectorization is enabled
Ganesha Shreedhara created HIVE-21428:
-
Summary: field delimiter of serde set for partition is not getting
respected when vectorization is enabled
Key: HIVE-21428
URL: https://issues.apache.org/jira/browse/HIVE-21428
Project: Hive
Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Ganesha Shreedhara
*Steps to reproduce:*
create external table src (c1 string, c2, string, c3 string) partitioned by
(part string)
location '/tmp/src';
echo "d1\td2" >> data.txt;
hadoop dfs -put data.txt /tmp/src/part=part1/;
MSCK REPAIR TABLE src;
ALTER TABLE src PARTITION (part='part1')
SET SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('columns'='c1,c2', 'column.types' ='string,string',
'field.delim'='\t');
create table dest (c1 string, c2 string, c3 string, c4 string);
insert overwrite table dest select * from src;
select * from dest;
*Result* (wrong)*:*
d1 d2 NULL NULL part1
set hive.vectorized.execution.enabled=false;
insert overwrite table dest select * from src;
select * from dest;
*Result* (Correct)*:*
d1 d2 NULL part1
This is because "d1\td2" is getting considered as single column because the
filed delimiter used by deserialiser is *^A* instead of *\t* which is set at
partition level.
It is working fine if I alter the field delimiter of serde for the entire table.
So, looks like serde properties in TableDesc is taking precedence over serde
properties in PartitionDesc.
This issue is not there in 2.x versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [hive] vineetgarg02 commented on a change in pull request #557: HIVE-21338 Remove order by and limit for aggregates
vineetgarg02 commented on a change in pull request #557: HIVE-21338 Remove order by and limit for aggregates URL: https://github.com/apache/hive/pull/557#discussion_r264425844 ## File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java ## @@ -1925,6 +1926,11 @@ public RelNode apply(RelOptCluster cluster, RelOptSchema relOptSchema, SchemaPlu perfLogger.PerfLogEnd(this.getClass().getName(), PerfLogger.OPTIMIZER, "Calcite: Window fixing rule"); } + perfLogger.PerfLogBegin(this.getClass().getName(), PerfLogger.OPTIMIZER); Review comment: @jcamachor I have moved the rule to `applyPreJoinOrderingTransforms` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HIVE-21427) Syslog serde
Ashutosh Chauhan created HIVE-21427: --- Summary: Syslog serde Key: HIVE-21427 URL: https://issues.apache.org/jira/browse/HIVE-21427 Project: Hive Issue Type: New Feature Components: Serializers/Deserializers Reporter: Ashutosh Chauhan It will be useful to read syslog generated log files in Hive -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21426) Remove Utilities Global Random
David Mollitor created HIVE-21426:
-
Summary: Remove Utilities Global Random
Key: HIVE-21426
URL: https://issues.apache.org/jira/browse/HIVE-21426
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Affects Versions: 4.0.0, 3.2.0
Reporter: David Mollitor
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L253
Remove global {{Random}} object in favor of {{ThreadLocalRandom}}.
{quote}
ThreadLocalRandom is initialized with an internally generated seed that may not
otherwise be modified. When applicable, use of ThreadLocalRandom rather than
shared Random objects in concurrent programs will typically encounter much less
overhead and contention.
{quote}
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadLocalRandom.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [hive] BELUGABEHR opened a new pull request #565: HIVE-21426: Remove Utilities Global Random
BELUGABEHR opened a new pull request #565: HIVE-21426: Remove Utilities Global Random URL: https://github.com/apache/hive/pull/565 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hive] maheshk114 commented on a change in pull request #562: HIVE-21421: HiveStatement.getQueryId throws NPE when query is not running.
maheshk114 commented on a change in pull request #562: HIVE-21421:
HiveStatement.getQueryId throws NPE when query is not running.
URL: https://github.com/apache/hive/pull/562#discussion_r264355254
##
File path: jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java
##
@@ -1007,8 +1007,17 @@ public void setInPlaceUpdateStream(InPlaceUpdateStream
stream) {
this.inPlaceUpdateStream = stream;
}
- @VisibleForTesting
+ /**
+ * Returns the Query ID if it is running.
+ * This method is a public API for usage outside of Hive, although it is not
part of the
+ * interface java.sql.Statement.
+ * @return Valid query ID if it is running else returns NULL.
+ * @throws SQLException If any internal failures.
+ */
public String getQueryId() throws SQLException {
+if (stmtHandle == null) {
+ return null;
Review comment:
what about concurrent closure of statement ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] BELUGABEHR opened a new pull request #564: HIVE-21425: Use DirectExecutorService for getInputSummary
BELUGABEHR opened a new pull request #564: HIVE-21425: Use DirectExecutorService for getInputSummary URL: https://github.com/apache/hive/pull/564 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HIVE-21425) Use newDirectExecutorService for getInputSummary
David Mollitor created HIVE-21425:
-
Summary: Use newDirectExecutorService for getInputSummary
Key: HIVE-21425
URL: https://issues.apache.org/jira/browse/HIVE-21425
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Affects Versions: 4.0.0, 3.2.0
Reporter: David Mollitor
{code:java|title=Utilities.java}
int numExecutors = getMaxExecutorsForInputListing(ctx.getConf(),
pathNeedProcess.size());
if (numExecutors > 1) {
LOG.info("Using {} threads for getContentSummary", numExecutors);
executor = Executors.newFixedThreadPool(numExecutors,
new ThreadFactoryBuilder().setDaemon(true)
.setNameFormat("Get-Input-Summary-%d").build());
} else {
executor = null;
}
{code}
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2482-L2490
Instead of using a 'null' {{ExecutorService}}, use Guava's
{{DirectExecutorService}} and remove special casing for a 'null' value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-21424) Disable AggregateStatsCache by default
Zoltan Haindrich created HIVE-21424: --- Summary: Disable AggregateStatsCache by default Key: HIVE-21424 URL: https://issues.apache.org/jira/browse/HIVE-21424 Project: Hive Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Avoiding merge commits
I'm +1 for avoiding merge commits on master and release branches. .. Owen On Mon, Mar 11, 2019 at 7:47 AM Ashutosh Chauhan wrote: > Hi, > With advent of gitbox/github integration, folks have started using merge > commits (inadvertently I believe by merging via Github). This causes issues > downstream where in my branch, I can't cherry-pick and rebase, but rather > get merge history. So, I propose we ban merge commits. > Too radical? > > Thanks, > Ashutosh >
Avoiding merge commits
Hi, With advent of gitbox/github integration, folks have started using merge commits (inadvertently I believe by merging via Github). This causes issues downstream where in my branch, I can't cherry-pick and rebase, but rather get merge history. So, I propose we ban merge commits. Too radical? Thanks, Ashutosh
[jira] [Created] (HIVE-21423) Do not check for whitespace issues in generated code
Miklos Gergely created HIVE-21423: - Summary: Do not check for whitespace issues in generated code Key: HIVE-21423 URL: https://issues.apache.org/jira/browse/HIVE-21423 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 3.1.1 Reporter: Miklos Gergely Assignee: Miklos Gergely Fix For: 4.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Review Request 70176: Break up DDLTask - extract Table related operations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70176/ --- Review request for hive and Zoltan Haindrich. Bugs: HIVE-21401 https://issues.apache.org/jira/browse/HIVE-21401 Repository: hive-git Description --- DDLTask is a huge class, more than 5000 lines long. The related DDLWork is also a huge class, which has a field for each DDL operation it supports. The goal is to refactor these in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: have a separate class for each operation have a package for each operation group (database ddl, table ddl, etc), so the amount of classes under a package is more manageable make all the requests (DDLDesc subclasses) immutable DDLTask should be agnostic to the actual operations right now let's ignore the issue of having some operations handled by DDLTask which are not actual DDL operations (lock, unlock, desc...) In the interim time when there are two DDLTask and DDLWork classes in the code base the new ones in the new package are called DDLTask2 and DDLWork2 thus avoiding the usage of fully qualified class names where both the old and the new classes are in use. Step #2: extract all the table related operations from the old DDLTask except alter table, and move them under the new package. Also updated the framework a bit, and fixed some smaller issue related to the previous step. Diffs - accumulo-handler/src/test/results/positive/accumulo_queries.q.out ac2d527 accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out ac809fa common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c33d03e contrib/src/test/results/clientnegative/serde_regex.q.out 58a4679 contrib/src/test/results/clientpositive/fileformat_base64.q.out 8e6a5e4 contrib/src/test/results/clientpositive/serde_regex.q.out 691e254 hbase-handler/src/test/results/positive/hbase_ddl.q.out e87240a hbase-handler/src/test/results/positive/hbase_queries.q.out 02f46d8 hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out b15515e hbase-handler/src/test/results/positive/hbasestats.q.out 5143522 hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateTableHook.java a377805 hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java fd159fe itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 45aac5f itests/util/src/main/java/org/apache/hadoop/hive/ql/metadata/DummySemanticAnalyzerHook.java 3575a16 itests/util/src/main/java/org/apache/hadoop/hive/ql/metadata/DummySemanticAnalyzerHook1.java e20ac64 ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLOperation.java e349a0a ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLOperationContext.java 924f0b3 ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLTask2.java 068e1e7 ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLWork2.java d2fbe8f ql/src/java/org/apache/hadoop/hive/ql/ddl/database/DescDatabaseOperation.java efaf389 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/ShowCreateDatabaseDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/database/ShowCreateDatabaseOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableLikeDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableLikeOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/CreateTableOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/DescTableDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/DescTableOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/DropTableDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/DropTableOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/LockTableDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/LockTableOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/PreInsertTableDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/PreInsertTableOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/ShowCreateTableDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/ShowCreateTableOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/ShowTablePropertiesDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/ShowTablePropertiesOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/ShowTableStatusDesc.java PRE-CREATION
[GitHub] [hive] odraese opened a new pull request #563: Hive 21422
odraese opened a new pull request #563: Hive 21422 URL: https://github.com/apache/hive/pull/563 https://issues.apache.org/jira/browse/HIVE-21422 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HIVE-21422) Add metrics to LRFU cache policy
Oliver Draese created HIVE-21422: Summary: Add metrics to LRFU cache policy Key: HIVE-21422 URL: https://issues.apache.org/jira/browse/HIVE-21422 Project: Hive Issue Type: Improvement Components: llap Affects Versions: 4.0.0 Reporter: Oliver Draese Assignee: Oliver Draese Fix For: 4.0.0 The LRFU cache policy for the LLAP data cache doesn't provide enough insight to figure out, what is cached and why something might get evicted. This ticket is used to add Hadoop metrics 2 information (accessible via JMX) to the LRFU policy, providing following information: * How much memory is cached for data buffers * How much memory is cached for meta data buffers * How large is the min-heap of the cache policy * How long is the eviction short list (linked list) * How much memory is currently "locked" (buffers with positive reference count) and therefore in use by a query These new counters are found in the MX bean, following this path: Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hive] sankarh opened a new pull request #562: HIVE-21421: HiveStatement.getQueryId throws NPE when query is not running.
sankarh opened a new pull request #562: HIVE-21421: HiveStatement.getQueryId throws NPE when query is not running. URL: https://github.com/apache/hive/pull/562 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HIVE-21421) HiveStatement.getQueryId throws NPE when query is not running.
Sankar Hariappan created HIVE-21421: --- Summary: HiveStatement.getQueryId throws NPE when query is not running. Key: HIVE-21421 URL: https://issues.apache.org/jira/browse/HIVE-21421 Project: Hive Issue Type: Bug Components: JDBC Affects Versions: 4.0.0 Reporter: Sankar Hariappan Assignee: Sankar Hariappan HiveStatement.getQueryId throws NullPointerException if it invoked without executing any query or query is closed. It should instead return null so that caller would check it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hive] miklosgergely commented on a change in pull request #543: HIVE-21292: Break up DDLTask 1 - extract Database related operations
miklosgergely commented on a change in pull request #543: HIVE-21292: Break up
DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264151355
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/database/CreateDatabaseOperation.java
##
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl.database;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.ddl.DDLOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState;
+
+/**
+ * Operation process of creating a database.
+ */
+public class CreateDatabaseOperation extends DDLOperation {
+ private static final String DATABASE_PATH_SUFFIX = ".db";
+
+ @Override
+ public int execute() throws HiveException {
+Database database = new Database();
+database.setName(ddlDesc.getName());
+database.setDescription(ddlDesc.getComment());
+database.setLocationUri(ddlDesc.getLocationUri());
+database.setParameters(ddlDesc.getDatabaseProperties());
+database.setOwnerName(SessionState.getUserFromAuthenticator());
+database.setOwnerType(PrincipalType.USER);
+
+try {
+ makeLocationQualified(database);
+ db.createDatabase(database, ddlDesc.getIfNotExists());
+} catch (AlreadyExistsException ex) {
+ //it would be better if AlreadyExistsException had an errorCode field
+ throw new HiveException(ex, ErrorMsg.DATABASE_ALREADY_EXISTS,
ddlDesc.getName());
+}
+
+return 0;
+ }
+
+ private void makeLocationQualified(Database database) throws HiveException {
Review comment:
I don't see how, the goal of this method is to make the location uri of the
database qualified, so Database needs to be passed to it. Or would you rather
have a method that just returns the new qualified locationUri, and then pass it
to the database?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] miklosgergely commented on a change in pull request #543: HIVE-21292: Break up DDLTask 1 - extract Database related operations
miklosgergely commented on a change in pull request #543: HIVE-21292: Break up
DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264150401
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/DDLTask2.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.hive.ql.CompilationOpContext;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryPlan;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.parse.ExplainConfiguration.AnalyzeState;
+import org.apache.hadoop.hive.ql.plan.api.StageType;
+
+/**
+ * DDLTask implementation.
+**/
+public class DDLTask2 extends Task implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private static final Map, Class>> DESC_TO_OPARATION =
+ new HashMap<>();
+ public static void registerOperator(Class descClass,
+ Class> operationClass) {
+DESC_TO_OPARATION.put(descClass, operationClass);
+ }
+
+ @Override
+ public boolean requireLock() {
+return this.work != null && this.work.getNeedLock();
+ }
+
+ @Override
+ public void initialize(QueryState queryState, QueryPlan queryPlan,
DriverContext ctx,
+ CompilationOpContext opContext) {
+super.initialize(queryState, queryPlan, ctx, opContext);
+ }
+
+ @Override
+ public int execute(DriverContext driverContext) {
+if (driverContext.getCtx().getExplainAnalyze() == AnalyzeState.RUNNING) {
+ return 0;
+}
+
+try {
+ Hive db = Hive.get(conf);
+ DDLDesc ddlDesc = work.getDDLDesc();
+
+ if (DESC_TO_OPARATION.containsKey(ddlDesc.getClass())) {
+DDLOperation ddlOperation =
DESC_TO_OPARATION.get(ddlDesc.getClass()).newInstance();
+ddlOperation.init(db, conf, driverContext, ddlDesc);
+return ddlOperation.execute();
+ } else {
+throw new IllegalArgumentException("Unknown DDL request: " +
ddlDesc.getClass());
+ }
+} catch (Throwable e) {
+ failed(e);
+ return 1;
Review comment:
Not all of them, any DDLOperation may return with an error code. Only those
cases when there is an exception is thrown we translate it to 1, as we have no
better option.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] miklosgergely commented on a change in pull request #543: HIVE-21292: Break up DDLTask 1 - extract Database related operations
miklosgergely commented on a change in pull request #543: HIVE-21292: Break up
DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264148937
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/database/DescDatabaseOperation.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl.database;
+
+import java.io.DataOutputStream;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.ddl.DDLOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Operation process of describing a database.
+ */
+public class DescDatabaseOperation extends DDLOperation {
+ @Override
+ public int execute() throws HiveException {
+DataOutputStream outStream = getOutputStream(ddlDesc.getResFile());
+try {
+ Database database = db.getDatabase(ddlDesc.getDatabaseName());
+ if (database == null) {
+throw new HiveException(ErrorMsg.DATABASE_NOT_EXISTS,
ddlDesc.getDatabaseName());
+ }
+
+ Map params = null;
+ if (ddlDesc.isExt()) {
+params = database.getParameters();
+ }
+
+ // If this is a q-test, let's order the params map (lexicographically) by
+ // key. This is to get consistent param ordering between Java7 and Java8.
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST) && params
!= null) {
+params = new TreeMap(params);
+ }
+
+ String location = database.getLocationUri();
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST)) {
+location = "location/in/test";
+ }
+
+ PrincipalType ownerType = database.getOwnerType();
+ formatter.showDatabaseDescription(outStream, database.getName(),
database.getDescription(), location,
+ database.getOwnerName(), (null == ownerType) ? null :
ownerType.name(), params);
Review comment:
Fixing in the next refactor in the DDL cleanup.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] miklosgergely commented on a change in pull request #543: HIVE-21292: Break up DDLTask 1 - extract Database related operations
miklosgergely commented on a change in pull request #543: HIVE-21292: Break up
DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264137620
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/database/DescDatabaseOperation.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl.database;
+
+import java.io.DataOutputStream;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.ddl.DDLOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Operation process of describing a database.
+ */
+public class DescDatabaseOperation extends DDLOperation {
+ @Override
+ public int execute() throws HiveException {
+DataOutputStream outStream = getOutputStream(ddlDesc.getResFile());
+try {
+ Database database = db.getDatabase(ddlDesc.getDatabaseName());
+ if (database == null) {
+throw new HiveException(ErrorMsg.DATABASE_NOT_EXISTS,
ddlDesc.getDatabaseName());
+ }
+
+ Map params = null;
+ if (ddlDesc.isExt()) {
+params = database.getParameters();
+ }
+
+ // If this is a q-test, let's order the params map (lexicographically) by
+ // key. This is to get consistent param ordering between Java7 and Java8.
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST) && params
!= null) {
+params = new TreeMap(params);
+ }
+
+ String location = database.getLocationUri();
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST)) {
+location = "location/in/test";
Review comment:
HIVE-21420 opened
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] miklosgergely commented on a change in pull request #543: HIVE-21292: Break up DDLTask 1 - extract Database related operations
miklosgergely commented on a change in pull request #543: HIVE-21292: Break up
DDLTask 1 - extract Database related operations
URL: https://github.com/apache/hive/pull/543#discussion_r264137508
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/ddl/database/DescDatabaseOperation.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.ddl.database;
+
+import java.io.DataOutputStream;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.PrincipalType;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.ddl.DDLOperation;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.io.IOUtils;
+
+/**
+ * Operation process of describing a database.
+ */
+public class DescDatabaseOperation extends DDLOperation {
+ @Override
+ public int execute() throws HiveException {
+DataOutputStream outStream = getOutputStream(ddlDesc.getResFile());
+try {
+ Database database = db.getDatabase(ddlDesc.getDatabaseName());
+ if (database == null) {
+throw new HiveException(ErrorMsg.DATABASE_NOT_EXISTS,
ddlDesc.getDatabaseName());
+ }
+
+ Map params = null;
+ if (ddlDesc.isExt()) {
+params = database.getParameters();
+ }
+
+ // If this is a q-test, let's order the params map (lexicographically) by
+ // key. This is to get consistent param ordering between Java7 and Java8.
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_IN_TEST) && params
!= null) {
+params = new TreeMap(params);
Review comment:
HIVE-21420 opened
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (HIVE-21420) DescDatabaseOperation should not handle test use cases
Miklos Gergely created HIVE-21420: - Summary: DescDatabaseOperation should not handle test use cases Key: HIVE-21420 URL: https://issues.apache.org/jira/browse/HIVE-21420 Project: Hive Issue Type: Improvement Reporter: Miklos Gergely [~kgyrtkirk]: DescDatabaseOperation should not be customized because we have testswe should have some other way to address this -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hive] sankarh closed pull request #551: HIVE-21286: Hive should support clean-up of previously bootstrapped tables when retry from different dump.
sankarh closed pull request #551: HIVE-21286: Hive should support clean-up of previously bootstrapped tables when retry from different dump. URL: https://github.com/apache/hive/pull/551 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #388: HIVE-20057: Fix Hive table conversion DESCRIBE table bug
sankarh commented on a change in pull request #388: HIVE-20057: Fix Hive table
conversion DESCRIBE table bug
URL: https://github.com/apache/hive/pull/388#discussion_r264106935
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -1802,12 +1802,12 @@ private MTable convertToMTable(Table tbl) throws
InvalidObjectException,
// accordingly
String tableType = tbl.getTableType();
boolean isExternal =
Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
-if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
+if (TableType.MANAGED_TABLE.toString().equalsIgnoreCase(tableType)) {
Review comment:
@animenon
Thanks for thr reply! But, I'm still confused.
As per your point, isExternal is proper (both "True" and "TRUE"). But, why
would tableType have different case based on input for EXTERNAL property?
Below is the code that reads EXTERNAL property from ALTER TABLE command and
sets correct tableType.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L4092
```
String externalProp = alterTbl.getProps().get("EXTERNAL");
if (externalProp != null) {
if (Boolean.parseBoolean(externalProp) && tbl.getTableType() ==
TableType.MANAGED_TABLE) {
tbl.setTableType(TableType.EXTERNAL_TABLE);
} else if (!Boolean.parseBoolean(externalProp) && tbl.getTableType()
== TableType.EXTERNAL_TABLE) {
tbl.setTableType(TableType.MANAGED_TABLE);
}
}
```
If you notice, we use Boolean.parseBoolean(externalProp) here which ignores
case.
I didn't get how two values of externalProp ("True" and "TRUE") influences
the table type set via setTableType.
Pls point to what I'm missing here.
@rmsmani
Please check if I miss anything.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #388: HIVE-20057: Fix Hive table conversion DESCRIBE table bug
sankarh commented on a change in pull request #388: HIVE-20057: Fix Hive table
conversion DESCRIBE table bug
URL: https://github.com/apache/hive/pull/388#discussion_r264106935
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
##
@@ -1802,12 +1802,12 @@ private MTable convertToMTable(Table tbl) throws
InvalidObjectException,
// accordingly
String tableType = tbl.getTableType();
boolean isExternal =
Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
-if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
+if (TableType.MANAGED_TABLE.toString().equalsIgnoreCase(tableType)) {
Review comment:
@animenon
Thanks for thr reply! But, I'm still confused.
As per your point, isExternal is proper (both "True" and "TRUE"). But, why
would tableType have different case based on input for EXTERNAL property?
Below is the code that reads EXTERNAL property from ALTER TABLE command and
sets correct tableType.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java#L4092
```
String externalProp = alterTbl.getProps().get("EXTERNAL");
if (externalProp != null) {
if (Boolean.parseBoolean(externalProp) && tbl.getTableType() ==
TableType.MANAGED_TABLE) {
tbl.setTableType(TableType.EXTERNAL_TABLE);
} else if (!Boolean.parseBoolean(externalProp) && tbl.getTableType()
== TableType.EXTERNAL_TABLE) {
tbl.setTableType(TableType.MANAGED_TABLE);
}
}
```
If you notice, we use Boolean.parseBoolean(externalProp) here which ignores
case.
I didn't get how two values of externalProp ("True" and "TRUE") influences
the table type set via setTableType.
Pls point to what I'm missing here.
@rmsmani
Please check if I miss anything.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
