[jira] [Created] (HIVE-21463) Table "partition_keys" has been specified with a primary-key to include column "TBL_ID"
yongjian.wu created HIVE-21463: -- Summary: Table "partition_keys" has been specified with a primary-key to include column "TBL_ID" Key: HIVE-21463 URL: https://issues.apache.org/jira/browse/HIVE-21463 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 2.3.4 Reporter: yongjian.wu Hi,when i use the Hive-2.3.4 with the mariadb10.2.14 as the mete data db,i meet the bellow error message: hive> create table jian(ii char(1)); FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Table "partition_keys" has been specified with a primary-key to include column "TBL_ID" but this column is not found in the table. Please check your column specification.) and about my mete data db you can see: 13:41:25 (root@localhost) [jian]> show create table partition_keys; | partition_keys | CREATE TABLE `partition_keys` ( `TBL_ID` bigint(20) NOT NULL, `PKEY_COMMENT` varchar(4000) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL, `PKEY_NAME` varchar(128) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL, `PKEY_TYPE` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL, `INTEGER_IDX` int(11) NOT NULL, PRIMARY KEY (`TBL_ID`,`PKEY_NAME`), KEY `PARTITION_KEYS_N49` (`TBL_ID`), CONSTRAINT `PARTITION_KEYS_FK1` FOREIGN KEY (`TBL_ID`) REFERENCES `tbls` (`TBL_ID`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 | when i create a database is can be working bu when create table it is error occur hive> create database jiantest; OK Time taken: 6.783 seconds hive> show databases; OK default jiantest Time taken: 0.236 seconds, Fetched: 2 row(s) this my config file if needed: [root@hadoop hive-2.3.4]# cat conf/hive-site.xml hive.metastore.local true javax.jdo.option.ConnectionURL jdbc:mysql://172.17.0.5:3306/jian?characterEncoding=latin1 javax.jdo.option.ConnectionDriverName org.mariadb.jdbc.Driver mariadb-java-client-2.4.0.jar javax.jdo.option.ConnectionUserName jian javax.jdo.option.ConnectionPassword 123456 hive.metastore.schema.verification false waiting for you reply,thank you -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hive] sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server going OOM during hive external table replications
sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server
going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266297902
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -136,7 +143,11 @@ private void doCopyRetry(FileSystem sourceFs,
List s
}
// looks like some network outrage, reset the file system object and
retry.
- FileSystem.closeAllForUGI(Utils.getUGI());
+ if (proxyUser == null) {
+FileSystem.closeAllForUGI(Utils.getUGI());
+ } else {
+FileSystem.closeAllForUGI(proxyUser);
+ }
sourceFs = pathList.get(0).getFileSystem(hiveConf);
destinationFs = destination.getFileSystem(hiveConf);
Review comment:
We need to propagate the destinationFs to caller as well as caller still
points to old file system object which is invalid. As caller uses this object
in a loop, it is necessary to propagate it back to caller.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server going OOM during hive external table replications
sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server
going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266296535
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
+if (!checkIfPathExist(sourcePath, proxyUser)) {
+ LOG.warn("Source path is missing. Ignoring exception.");
Review comment:
Can be just info log as it is possible valid scenario in case of external
tables.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server going OOM during hive external table replications
sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server
going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266298085
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -72,33 +72,40 @@ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf)
{
public void copyAndVerify(FileSystem destinationFs, Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
-for (Map.Entry>>
entry : map.entrySet()) {
- FileSystem sourceFs = entry.getKey();
- Map> destMap = entry.getValue();
- for (Map.Entry> destMapEntry :
destMap.entrySet()) {
-Path destination = destMapEntry.getKey();
-List fileInfoList =
destMapEntry.getValue();
-boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
-
-if (!destinationFs.exists(destination)
-&& !FileUtils.mkdir(destinationFs, destination, hiveConf)) {
- LOG.error("Failed to create destination directory: " + destination);
- throw new IOException("Destination directory creation failed");
-}
+UserGroupInformation proxyUser = getProxyUser();
+try {
+ for (Map.Entry>>
entry : map.entrySet()) {
+FileSystem sourceFs = entry.getKey();
Review comment:
The sourceFs should be re-get from HDFS if previous copy closed all file
systems object for proxy user.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server going OOM during hive external table replications
sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server
going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266296453
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
Review comment:
It can be just info or debug.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hive] sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server going OOM during hive external table replications
sankarh commented on a change in pull request #569: HIVE-21446 : Hive Server
going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266297006
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -238,23 +249,54 @@ private boolean isSourceFileMismatch(FileSystem
sourceFs, ReplChangeManager.File
return false;
}
+ private UserGroupInformation getProxyUser() throws LoginException,
IOException {
+if (copyAsUser == null) {
+ return null;
+}
+UserGroupInformation proxyUser = null;
+int currentRetry = 0;
+while (currentRetry <= MAX_IO_RETRY) {
+ try {
+UserGroupInformation ugi = Utils.getUGI();
+String currentUser = ugi.getShortUserName();
+if (!currentUser.equals(copyAsUser)) {
+ proxyUser = UserGroupInformation.createProxyUser(
+ copyAsUser, UserGroupInformation.getLoginUser());
+}
+return proxyUser;
+ } catch (IOException e) {
+currentRetry++;
+if (currentRetry <= MAX_IO_RETRY) {
+ LOG.warn("Unable to get UGI info", e);
Review comment:
Can log the retry number in the log msg.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
Ashutosh Bapat created HIVE-21462: - Summary: Upgrading SQL server backed metastore when changing data type of a column with constraints Key: HIVE-21462 URL: https://issues.apache.org/jira/browse/HIVE-21462 Project: Hive Issue Type: Bug Components: Standalone Metastore Reporter: Ashutosh Bapat Assignee: Ashutosh Bapat Fix For: 4.0.0 SQL server does not allow changing data type of a column which has a constraint or an index on it. The constraint or the index needs to be dropped before changing the data type and needs to be recreated after that. Metastore upgrade scripts aren't doing this and thus upgrade fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Is Github PR mandatory?
Got it. Thanks! On Fri, Mar 15, 2019 at 1:47 PM Mani M wrote: > It's used as substitution for review board. > > > With Regards > M.Mani > +61 432 461 087 > > On Fri, 15 Mar 2019, 13:59 Rui Li, wrote: > > > Hi, > > > > I believe we still need to upload patch to JIRA for precommit testing. So > > just want to make sure whether opening a github PR is mandatory? Or is it > > just a substitution for the review board? > > > > -- > > Best regards! > > Rui Li > > > -- Best regards! Rui Li
[jira] [Created] (HIVE-21461) Query Cache Contains Incorrect Result For Virtual Columns
Shawn Weeks created HIVE-21461:
--
Summary: Query Cache Contains Incorrect Result For Virtual Columns
Key: HIVE-21461
URL: https://issues.apache.org/jira/browse/HIVE-21461
Project: Hive
Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Shawn Weeks
Hive query cache will contain incorrect results if query containing virtual
columns input__file__name or block__offset__inside__file are run prior to
compactions and then run immediately after. Not sure if these should be treated
like non-deterministic functions or if it's possible for Hive to know that the
underlying data has been rearranged due to compaction.
{code:java}
select count(*), input__file__name
from test_hive_streaming
group by input__file__name;
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
