[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=323197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323197
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 04/Oct/19 05:35
Start Date: 04/Oct/19 05:35
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791#discussion_r330946090
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TextInputFormat;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * SkippingInputFormat is a header/footer aware input format. It truncates
+ * splits identified by TextInputFormat. Header and footers are removed
+ * from the splits.
+ */
+public class SkippingTextInputFormat extends TextInputFormat {
+
+ private final Map startIndexMap = new ConcurrentHashMap();
+ private final Map endIndexMap = new ConcurrentHashMap();
+ private JobConf conf;
+ private int headerCount;
+ private int footerCount;
+
+ @Override
+ public void configure(JobConf conf) {
+this.conf = conf;
+super.configure(conf);
+ }
+
+ public void configure(JobConf conf, int headerCount, int footerCount) {
+configure(conf);
+this.headerCount = headerCount;
+this.footerCount = footerCount;
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts) {
+return makeSplitInternal(file, start, length, hosts, null);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts, String[] inMemoryHosts) {
+return makeSplitInternal(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit makeSplitInternal(Path file, long start, long length,
String[] hosts, String[] inMemoryHosts) {
+long cachedStart;
+long cachedEnd;
+try {
+ cachedStart = getCachedStartIndex(file);
+ cachedEnd = getCachedEndIndex(file);
+} catch (IOException e) {
+ LOG.warn("Could not detect header/footer", e);
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start + length) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start) {
+ length = length - (cachedStart - start);
+ start = cachedStart;
+}
+if (cachedEnd < start) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedEnd < start + length) {
+ length = cachedEnd - start;
+}
+if (inMemoryHosts == null) {
+ return super.makeSplit(file, start, length, hosts);
+} else {
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+}
+ }
+
+ private long getCachedStartIndex(Path path) throws IOException {
+if (headerCount == 0) {
+ return 0;
+}
+Long startIndexForFile = startIndexMap.get(path);
+if (startIndexForFile == null) {
+ FileSystem fileSystem;
+ FSDataInputStream fis = null;
+ fileSystem = path.getFileSystem(conf);
+ try {
+fis = fileSystem.open(path);
+for (int j = 0; j < headerCount; j++) {
+ if (fis.readLine() == null) {
+startIndexMap.put(path, Long.MAX_VALUE);
+return Long.MAX_VALUE;
+ }
+}
+// back 1 byte because readers skip the entire first row if split
start is not 0
+startIndexForFile = fis.getPos() - 1;
Review comment:
I'm not sure about this. My understanding is that getPos() returns position
of first character in the line. Do we need the -1?
[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=323194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323194
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 04/Oct/19 05:33
Start Date: 04/Oct/19 05:33
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791#discussion_r331338984
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TextInputFormat;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * SkippingInputFormat is a header/footer aware input format. It truncates
+ * splits identified by TextInputFormat. Header and footers are removed
+ * from the splits.
+ */
+public class SkippingTextInputFormat extends TextInputFormat {
+
+ private final Map startIndexMap = new ConcurrentHashMap();
+ private final Map endIndexMap = new ConcurrentHashMap();
+ private JobConf conf;
+ private int headerCount;
+ private int footerCount;
+
+ @Override
+ public void configure(JobConf conf) {
+this.conf = conf;
+super.configure(conf);
+ }
+
+ public void configure(JobConf conf, int headerCount, int footerCount) {
+configure(conf);
+this.headerCount = headerCount;
+this.footerCount = footerCount;
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts) {
+return makeSplitInternal(file, start, length, hosts, null);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts, String[] inMemoryHosts) {
+return makeSplitInternal(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit makeSplitInternal(Path file, long start, long length,
String[] hosts, String[] inMemoryHosts) {
+long cachedStart;
+long cachedEnd;
+try {
+ cachedStart = getCachedStartIndex(file);
+ cachedEnd = getCachedEndIndex(file);
+} catch (IOException e) {
+ LOG.warn("Could not detect header/footer", e);
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start + length) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start) {
+ length = length - (cachedStart - start);
+ start = cachedStart;
+}
+if (cachedEnd < start) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedEnd < start + length) {
+ length = cachedEnd - start;
+}
+if (inMemoryHosts == null) {
+ return super.makeSplit(file, start, length, hosts);
+} else {
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+}
+ }
+
+ private long getCachedStartIndex(Path path) throws IOException {
+if (headerCount == 0) {
+ return 0;
+}
+Long startIndexForFile = startIndexMap.get(path);
+if (startIndexForFile == null) {
+ FileSystem fileSystem;
+ FSDataInputStream fis = null;
+ fileSystem = path.getFileSystem(conf);
+ try {
+fis = fileSystem.open(path);
+for (int j = 0; j < headerCount; j++) {
+ if (fis.readLine() == null) {
+startIndexMap.put(path, Long.MAX_VALUE);
+return Long.MAX_VALUE;
+ }
+}
+// back 1 byte because readers skip the entire first row if split
start is not 0
+startIndexForFile = fis.getPos() - 1;
+ } finally {
+if (fis != null) {
+ fis.close();
+}
+ }
+ startIndexMap.put(path, startIndexForFile);
+}
+return startIndexForFile;
+ }
[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=323195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323195
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 04/Oct/19 05:33
Start Date: 04/Oct/19 05:33
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791#discussion_r331345123
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TextInputFormat;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * SkippingInputFormat is a header/footer aware input format. It truncates
+ * splits identified by TextInputFormat. Header and footers are removed
+ * from the splits.
+ */
+public class SkippingTextInputFormat extends TextInputFormat {
+
+ private final Map startIndexMap = new ConcurrentHashMap();
+ private final Map endIndexMap = new ConcurrentHashMap();
+ private JobConf conf;
+ private int headerCount;
+ private int footerCount;
+
+ @Override
+ public void configure(JobConf conf) {
+this.conf = conf;
+super.configure(conf);
+ }
+
+ public void configure(JobConf conf, int headerCount, int footerCount) {
+configure(conf);
+this.headerCount = headerCount;
+this.footerCount = footerCount;
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts) {
+return makeSplitInternal(file, start, length, hosts, null);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts, String[] inMemoryHosts) {
+return makeSplitInternal(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit makeSplitInternal(Path file, long start, long length,
String[] hosts, String[] inMemoryHosts) {
+long cachedStart;
+long cachedEnd;
+try {
+ cachedStart = getCachedStartIndex(file);
+ cachedEnd = getCachedEndIndex(file);
+} catch (IOException e) {
+ LOG.warn("Could not detect header/footer", e);
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start + length) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start) {
+ length = length - (cachedStart - start);
+ start = cachedStart;
+}
+if (cachedEnd < start) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedEnd < start + length) {
+ length = cachedEnd - start;
+}
+if (inMemoryHosts == null) {
+ return super.makeSplit(file, start, length, hosts);
+} else {
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+}
+ }
+
+ private long getCachedStartIndex(Path path) throws IOException {
+if (headerCount == 0) {
+ return 0;
+}
+Long startIndexForFile = startIndexMap.get(path);
+if (startIndexForFile == null) {
+ FileSystem fileSystem;
+ FSDataInputStream fis = null;
+ fileSystem = path.getFileSystem(conf);
+ try {
+fis = fileSystem.open(path);
+for (int j = 0; j < headerCount; j++) {
+ if (fis.readLine() == null) {
+startIndexMap.put(path, Long.MAX_VALUE);
+return Long.MAX_VALUE;
+ }
+}
+// back 1 byte because readers skip the entire first row if split
start is not 0
+startIndexForFile = fis.getPos() - 1;
+ } finally {
+if (fis != null) {
+ fis.close();
+}
+ }
+ startIndexMap.put(path, startIndexForFile);
+}
+return startIndexForFile;
+ }
[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=323191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323191
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 04/Oct/19 05:33
Start Date: 04/Oct/19 05:33
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791#discussion_r330946090
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TextInputFormat;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * SkippingInputFormat is a header/footer aware input format. It truncates
+ * splits identified by TextInputFormat. Header and footers are removed
+ * from the splits.
+ */
+public class SkippingTextInputFormat extends TextInputFormat {
+
+ private final Map startIndexMap = new ConcurrentHashMap();
+ private final Map endIndexMap = new ConcurrentHashMap();
+ private JobConf conf;
+ private int headerCount;
+ private int footerCount;
+
+ @Override
+ public void configure(JobConf conf) {
+this.conf = conf;
+super.configure(conf);
+ }
+
+ public void configure(JobConf conf, int headerCount, int footerCount) {
+configure(conf);
+this.headerCount = headerCount;
+this.footerCount = footerCount;
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts) {
+return makeSplitInternal(file, start, length, hosts, null);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts, String[] inMemoryHosts) {
+return makeSplitInternal(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit makeSplitInternal(Path file, long start, long length,
String[] hosts, String[] inMemoryHosts) {
+long cachedStart;
+long cachedEnd;
+try {
+ cachedStart = getCachedStartIndex(file);
+ cachedEnd = getCachedEndIndex(file);
+} catch (IOException e) {
+ LOG.warn("Could not detect header/footer", e);
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start + length) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start) {
+ length = length - (cachedStart - start);
+ start = cachedStart;
+}
+if (cachedEnd < start) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedEnd < start + length) {
+ length = cachedEnd - start;
+}
+if (inMemoryHosts == null) {
+ return super.makeSplit(file, start, length, hosts);
+} else {
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+}
+ }
+
+ private long getCachedStartIndex(Path path) throws IOException {
+if (headerCount == 0) {
+ return 0;
+}
+Long startIndexForFile = startIndexMap.get(path);
+if (startIndexForFile == null) {
+ FileSystem fileSystem;
+ FSDataInputStream fis = null;
+ fileSystem = path.getFileSystem(conf);
+ try {
+fis = fileSystem.open(path);
+for (int j = 0; j < headerCount; j++) {
+ if (fis.readLine() == null) {
+startIndexMap.put(path, Long.MAX_VALUE);
+return Long.MAX_VALUE;
+ }
+}
+// back 1 byte because readers skip the entire first row if split
start is not 0
+startIndexForFile = fis.getPos() - 1;
Review comment:
I'm not sure about this. My understanding is that getPos() returns position
of first character in the row. Do we need the -1?
[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=323192=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323192
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 04/Oct/19 05:33
Start Date: 04/Oct/19 05:33
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791#discussion_r331344762
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TextInputFormat;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * SkippingInputFormat is a header/footer aware input format. It truncates
+ * splits identified by TextInputFormat. Header and footers are removed
+ * from the splits.
+ */
+public class SkippingTextInputFormat extends TextInputFormat {
+
+ private final Map startIndexMap = new ConcurrentHashMap();
+ private final Map endIndexMap = new ConcurrentHashMap();
+ private JobConf conf;
+ private int headerCount;
+ private int footerCount;
+
+ @Override
+ public void configure(JobConf conf) {
+this.conf = conf;
+super.configure(conf);
+ }
+
+ public void configure(JobConf conf, int headerCount, int footerCount) {
+configure(conf);
+this.headerCount = headerCount;
+this.footerCount = footerCount;
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts) {
+return makeSplitInternal(file, start, length, hosts, null);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts, String[] inMemoryHosts) {
+return makeSplitInternal(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit makeSplitInternal(Path file, long start, long length,
String[] hosts, String[] inMemoryHosts) {
+long cachedStart;
+long cachedEnd;
+try {
+ cachedStart = getCachedStartIndex(file);
+ cachedEnd = getCachedEndIndex(file);
+} catch (IOException e) {
+ LOG.warn("Could not detect header/footer", e);
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start + length) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start) {
+ length = length - (cachedStart - start);
+ start = cachedStart;
+}
+if (cachedEnd < start) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedEnd < start + length) {
+ length = cachedEnd - start;
+}
+if (inMemoryHosts == null) {
+ return super.makeSplit(file, start, length, hosts);
+} else {
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+}
+ }
+
+ private long getCachedStartIndex(Path path) throws IOException {
+if (headerCount == 0) {
+ return 0;
+}
+Long startIndexForFile = startIndexMap.get(path);
+if (startIndexForFile == null) {
+ FileSystem fileSystem;
+ FSDataInputStream fis = null;
+ fileSystem = path.getFileSystem(conf);
+ try {
+fis = fileSystem.open(path);
+for (int j = 0; j < headerCount; j++) {
+ if (fis.readLine() == null) {
+startIndexMap.put(path, Long.MAX_VALUE);
+return Long.MAX_VALUE;
+ }
+}
+// back 1 byte because readers skip the entire first row if split
start is not 0
+startIndexForFile = fis.getPos() - 1;
+ } finally {
+if (fis != null) {
+ fis.close();
+}
+ }
+ startIndexMap.put(path, startIndexForFile);
+}
+return startIndexForFile;
+ }
[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=323196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323196
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 04/Oct/19 05:33
Start Date: 04/Oct/19 05:33
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791#discussion_r331344245
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TextInputFormat;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * SkippingInputFormat is a header/footer aware input format. It truncates
+ * splits identified by TextInputFormat. Header and footers are removed
+ * from the splits.
+ */
+public class SkippingTextInputFormat extends TextInputFormat {
+
+ private final Map startIndexMap = new ConcurrentHashMap();
+ private final Map endIndexMap = new ConcurrentHashMap();
+ private JobConf conf;
+ private int headerCount;
+ private int footerCount;
+
+ @Override
+ public void configure(JobConf conf) {
+this.conf = conf;
+super.configure(conf);
+ }
+
+ public void configure(JobConf conf, int headerCount, int footerCount) {
+configure(conf);
+this.headerCount = headerCount;
+this.footerCount = footerCount;
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts) {
+return makeSplitInternal(file, start, length, hosts, null);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts, String[] inMemoryHosts) {
+return makeSplitInternal(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit makeSplitInternal(Path file, long start, long length,
String[] hosts, String[] inMemoryHosts) {
+long cachedStart;
+long cachedEnd;
+try {
+ cachedStart = getCachedStartIndex(file);
+ cachedEnd = getCachedEndIndex(file);
+} catch (IOException e) {
+ LOG.warn("Could not detect header/footer", e);
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start + length) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start) {
+ length = length - (cachedStart - start);
+ start = cachedStart;
+}
+if (cachedEnd < start) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedEnd < start + length) {
+ length = cachedEnd - start;
+}
+if (inMemoryHosts == null) {
+ return super.makeSplit(file, start, length, hosts);
+} else {
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+}
+ }
+
+ private long getCachedStartIndex(Path path) throws IOException {
+if (headerCount == 0) {
+ return 0;
+}
+Long startIndexForFile = startIndexMap.get(path);
+if (startIndexForFile == null) {
+ FileSystem fileSystem;
+ FSDataInputStream fis = null;
+ fileSystem = path.getFileSystem(conf);
+ try {
+fis = fileSystem.open(path);
+for (int j = 0; j < headerCount; j++) {
+ if (fis.readLine() == null) {
+startIndexMap.put(path, Long.MAX_VALUE);
+return Long.MAX_VALUE;
+ }
+}
+// back 1 byte because readers skip the entire first row if split
start is not 0
+startIndexForFile = fis.getPos() - 1;
+ } finally {
+if (fis != null) {
+ fis.close();
+}
+ }
+ startIndexMap.put(path, startIndexForFile);
+}
+return startIndexForFile;
+ }
[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=323193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323193
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 04/Oct/19 05:33
Start Date: 04/Oct/19 05:33
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791#discussion_r331344703
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TextInputFormat;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * SkippingInputFormat is a header/footer aware input format. It truncates
+ * splits identified by TextInputFormat. Header and footers are removed
+ * from the splits.
+ */
+public class SkippingTextInputFormat extends TextInputFormat {
+
+ private final Map startIndexMap = new ConcurrentHashMap();
+ private final Map endIndexMap = new ConcurrentHashMap();
+ private JobConf conf;
+ private int headerCount;
+ private int footerCount;
+
+ @Override
+ public void configure(JobConf conf) {
+this.conf = conf;
+super.configure(conf);
+ }
+
+ public void configure(JobConf conf, int headerCount, int footerCount) {
+configure(conf);
+this.headerCount = headerCount;
+this.footerCount = footerCount;
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts) {
+return makeSplitInternal(file, start, length, hosts, null);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts, String[] inMemoryHosts) {
+return makeSplitInternal(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit makeSplitInternal(Path file, long start, long length,
String[] hosts, String[] inMemoryHosts) {
+long cachedStart;
+long cachedEnd;
+try {
+ cachedStart = getCachedStartIndex(file);
+ cachedEnd = getCachedEndIndex(file);
+} catch (IOException e) {
+ LOG.warn("Could not detect header/footer", e);
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start + length) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start) {
+ length = length - (cachedStart - start);
+ start = cachedStart;
+}
+if (cachedEnd < start) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedEnd < start + length) {
+ length = cachedEnd - start;
+}
+if (inMemoryHosts == null) {
+ return super.makeSplit(file, start, length, hosts);
+} else {
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+}
+ }
+
+ private long getCachedStartIndex(Path path) throws IOException {
+if (headerCount == 0) {
+ return 0;
+}
+Long startIndexForFile = startIndexMap.get(path);
+if (startIndexForFile == null) {
+ FileSystem fileSystem;
+ FSDataInputStream fis = null;
+ fileSystem = path.getFileSystem(conf);
+ try {
+fis = fileSystem.open(path);
+for (int j = 0; j < headerCount; j++) {
+ if (fis.readLine() == null) {
+startIndexMap.put(path, Long.MAX_VALUE);
+return Long.MAX_VALUE;
+ }
+}
+// back 1 byte because readers skip the entire first row if split
start is not 0
+startIndexForFile = fis.getPos() - 1;
+ } finally {
+if (fis != null) {
+ fis.close();
+}
+ }
+ startIndexMap.put(path, startIndexForFile);
+}
+return startIndexForFile;
+ }
[jira] [Commented] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision
[
https://issues.apache.org/jira/browse/HIVE-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944237#comment-16944237
]
Hive QA commented on HIVE-14302:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982169/HIVE-14302.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17171 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_decimal_vectorized]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=63)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=177)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=139)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18862/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18862/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18862/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982169 - PreCommit-HIVE-Build
> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-14302.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Decimal support in the optimized hashtable was decided on the basis of the
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
> TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
> predicate: (a is not null and true) (type: boolean)
> Select Operator (SEL_2)
> expressions: a (type: decimal(10,2))
> outputColumnNames: _col0
> Reduce Output Operator (RS_6)
> key expressions: _col0 (type: decimal(11,2))
> sort order: +
> Map-reduce partition columns: _col0 (type: decimal(11,2))
> Join Operator (JOIN_8)
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col0 (type: decimal(11,2))
> 1 _col0 (type: decimal(11,2))
> Group By Operator (GBY_11)
> aggregations: count(1)
> mode: hash
> outputColumnNames: _col0
> {code}
> See cast up to Decimal(11, 2) in the plan, which normalizes both sides of the
> join to be able to compare HiveDecimal as-is.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-20150) TopNKey pushdown
[ https://issues.apache.org/jira/browse/HIVE-20150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20150: -- Assignee: Krisztian Kasa (was: Richard Zhang) > TopNKey pushdown > > > Key: HIVE-20150 > URL: https://issues.apache.org/jira/browse/HIVE-20150 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Teddy Choi >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20150.1.patch, HIVE-20150.10.patch, > HIVE-20150.11.patch, HIVE-20150.11.patch, HIVE-20150.2.patch, > HIVE-20150.4.patch, HIVE-20150.5.patch, HIVE-20150.6.patch, > HIVE-20150.7.patch, HIVE-20150.8.patch, HIVE-20150.9.patch > > > TopNKey operator is implemented in HIVE-17896, but it needs more work in > pushdown implementation. So this issue covers TopNKey pushdown implementation > with proper tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision
[
https://issues.apache.org/jira/browse/HIVE-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944207#comment-16944207
]
Hive QA commented on HIVE-14302:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18862/dev-support/hive-personality.sh
|
| git revision | master / 8bd0e2a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18862/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-14302.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Decimal support in the optimized hashtable was decided on the basis of the
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
> TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
> predicate: (a is not null and true) (type: boolean)
> Select Operator (SEL_2)
> expressions: a (type: decimal(10,2))
> outputColumnNames: _col0
> Reduce Output Operator (RS_6)
> key expressions: _col0 (type: decimal(11,2))
> sort order: +
> Map-reduce partition columns: _col0 (type: decimal(11,2))
> Join Operator (JOIN_8)
> condition map:
> Inner Join 0 to 1
>
[jira] [Commented] (HIVE-22282) Obtain LLAP delegation token only when LLAP is configured for Kerberos authentication
[
https://issues.apache.org/jira/browse/HIVE-22282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944200#comment-16944200
]
Hive QA commented on HIVE-22282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982113/HIVE-22282.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17170 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18861/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18861/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18861/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982113 - PreCommit-HIVE-Build
> Obtain LLAP delegation token only when LLAP is configured for Kerberos
> authentication
> -
>
> Key: HIVE-22282
> URL: https://issues.apache.org/jira/browse/HIVE-22282
> Project: Hive
> Issue Type: Improvement
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22282.1.patch, HIVE-22282.2.patch,
> HIVE-22282.3.patch
>
>
> Contains also Kerberos related Zookeeper configuration changes after refactor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22282) Obtain LLAP delegation token only when LLAP is configured for Kerberos authentication
[
https://issues.apache.org/jira/browse/HIVE-22282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944191#comment-16944191
]
Hive QA commented on HIVE-22282:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
26s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
7s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18861/dev-support/hive-personality.sh
|
| git revision | master / 8bd0e2a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common common llap-client
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18861/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Obtain LLAP delegation token only when LLAP is configured for Kerberos
> authentication
> -
>
> Key: HIVE-22282
> URL: https://issues.apache.org/jira/browse/HIVE-22282
> Project: Hive
> Issue Type: Improvement
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22282.1.patch, HIVE-22282.2.patch,
> HIVE-22282.3.patch
>
>
> Contains also Kerberos related Zookeeper configuration changes after
[jira] [Commented] (HIVE-20150) TopNKey pushdown
[
https://issues.apache.org/jira/browse/HIVE-20150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944173#comment-16944173
]
Hive QA commented on HIVE-20150:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12941630/HIVE-20150.11.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18860/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18860/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18860/
Messages:
{noformat}
This message was trimmed, see log for full details
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/llap/limit_pushdown.q.out:421
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/limit_pushdown.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out:450
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out' cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out:425
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out:237
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/orc_struct_type_vectorization.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out:213
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/parquet_complex_types_vectorization.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out:229
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/parquet_map_type_vectorization.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out:237
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/parquet_struct_type_vectorization.q.out'
with conflicts.
error: patch failed: ql/src/test/results/clientpositive/llap/topnkey.q.out:130
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/topnkey.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out:64
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/llap/vector_string_concat.q.out:351
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/vector_string_concat.q.out' with
conflicts.
error: patch failed:
ql/src/test/results/clientpositive/llap/vector_topnkey.q.out:30
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/vector_topnkey.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query10.q.out:172
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query10.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query15.q.out:76
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query15.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query17.q.out:143
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query17.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query27.q.out:90
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query27.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query35.q.out:168
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query35.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query40.q.out:97
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query40.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/perf/tez/query43.q.out:72
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/perf/tez/query43.q.out'
with conflicts.
error: patch failed:
[jira] [Commented] (HIVE-22270) Upgrade commons-io to 2.6
[
https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944171#comment-16944171
]
Hive QA commented on HIVE-22270:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982112/HIVE-22270.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17170 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18859/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18859/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18859/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982112 - PreCommit-HIVE-Build
> Upgrade commons-io to 2.6
> -
>
> Key: HIVE-22270
> URL: https://issues.apache.org/jira/browse/HIVE-22270
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22270.01.patch, HIVE-22270.01.patch,
> HIVE-22270.patch, HIVE-22270.patch, HIVE-22270.patch, HIVE-22270.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using commons-io 2.4 and according to HIVE-21273, a number
> of issues are present in it, which can be resolved by upgrading to 2.6:
> IOUtils copyLarge() and skip() methods are performance hogs
> affectsVersions:2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues]
> CharSequenceInputStream#reset() behaves incorrectly in case when buffer size
> is not dividable by data size
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues]
> [Tailer] InterruptedException while the thead is sleeping is silently ignored
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues]
> IOUtils.contentEquals* methods returns false if input1 == input2; should
> return true
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues]
> Apache Commons - standard links for documents are failing
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues]
> FileUtils.sizeOfDirectoryAsBigInteger can overflow
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues]
> Regression in FileUtils.readFileToString from 2.0.1
> affectsVersions:2.1;2.2;2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-453?filter=allopenissues]
> Correct exception message in FileUtils.getFile(File; String...)
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-479?filter=allopenissues]
> org.apache.commons.io.FileUtils#waitFor waits too long
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-481?filter=allopenissues]
> FilenameUtils should handle embedded null bytes
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-484?filter=allopenissues]
> Exceptions are suppressed incorrectly when copying files.
> affectsVersions:2.4;2.5
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-502?filter=allopenissues]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22275) OperationManager.queryIdOperation does not properly clean up multiple queryIds
[
https://issues.apache.org/jira/browse/HIVE-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944167#comment-16944167
]
Prasanth Jayachandran commented on HIVE-22275:
--
lgtm, +1. Looks like when the cleanup happens the queryId from session conf is
retrieved which will only reflect the last query id and ends up leaking all the
query ids before it.
> OperationManager.queryIdOperation does not properly clean up multiple queryIds
> --
>
> Key: HIVE-22275
> URL: https://issues.apache.org/jira/browse/HIVE-22275
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-22275.1.patch, HIVE-22275.2.patch
>
>
> In the case that multiple statements are run by a single Session before being
> cleaned up, it appears that OperationManager.queryIdOperation is not cleaned
> up properly.
> See the log statements below - with the exception of the first "Removed
> queryId:" log line, the queryId listed during cleanup is the same, when each
> of these handles should have their own queryId. Looks like only the last
> queryId executed is being cleaned up.
> As a result, HS2 can run out of memory as OperationManager.queryIdOperation
> grows and never cleans these queryIds/Operations up.
> {noformat}
> 2019-09-13T08:37:36,785 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9]
> 2019-09-13T08:37:38,432 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083736_c49cf3cc-cfe8-48a1-bd22-8b924dfb0396
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9] with tag: null
> 2019-09-13T08:37:38,469 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=24d0030c-0e49-45fb-a918-2276f0941cfb]
> 2019-09-13T08:37:52,662 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=b983802c-1dec-4fa0-8680-d05ab555321b]
> 2019-09-13T08:37:56,239 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=75dbc531-2964-47b2-84d7-85b59f88999c]
> 2019-09-13T08:38:02,551 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=72c79076-9d67-4894-a526-c233fa5450b2]
> 2019-09-13T08:38:10,558 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=17b30a62-612d-4b70-9ba7-4287d2d9229b]
> 2019-09-13T08:38:16,930 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=ea97e99d-cc77-470b-b49a-b869c73a4615]
> 2019-09-13T08:38:20,440 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=a277b789-ebb8-4925-878f-6728d3e8c5fb]
> 2019-09-13T08:38:26,303 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=9a023ab8-aa80-45db-af88-94790cc83033]
> 2019-09-13T08:38:30,791 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=b697c801-7da0-4544-bcfa-442eb1d3bd77]
> 2019-09-13T08:39:10,187 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=bda93c8f-0822-4592-a61c-4701720a1a5c]
> 2019-09-13T08:39:15,471 INFO
[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=323143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323143
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 04/Oct/19 02:08
Start Date: 04/Oct/19 02:08
Worklog Time Spent: 10m
Work Description: jdere commented on pull request #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791#discussion_r331316286
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/io/SkippingTextInputFormat.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TextInputFormat;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Map;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * SkippingInputFormat is a header/footer aware input format. It truncates
+ * splits identified by TextInputFormat. Header and footers are removed
+ * from the splits.
+ */
+public class SkippingTextInputFormat extends TextInputFormat {
+
+ private final Map startIndexMap = new ConcurrentHashMap();
+ private final Map endIndexMap = new ConcurrentHashMap();
+ private JobConf conf;
+ private int headerCount;
+ private int footerCount;
+
+ @Override
+ public void configure(JobConf conf) {
+this.conf = conf;
+super.configure(conf);
+ }
+
+ public void configure(JobConf conf, int headerCount, int footerCount) {
+configure(conf);
+this.headerCount = headerCount;
+this.footerCount = footerCount;
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts) {
+return makeSplitInternal(file, start, length, hosts, null);
+ }
+
+ @Override
+ protected FileSplit makeSplit(Path file, long start, long length, String[]
hosts, String[] inMemoryHosts) {
+return makeSplitInternal(file, start, length, hosts, inMemoryHosts);
+ }
+
+ private FileSplit makeSplitInternal(Path file, long start, long length,
String[] hosts, String[] inMemoryHosts) {
+long cachedStart;
+long cachedEnd;
+try {
+ cachedStart = getCachedStartIndex(file);
+ cachedEnd = getCachedEndIndex(file);
+} catch (IOException e) {
+ LOG.warn("Could not detect header/footer", e);
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start + length) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedStart > start) {
+ length = length - (cachedStart - start);
+ start = cachedStart;
+}
+if (cachedEnd < start) {
+ return new NullRowsInputFormat.DummyInputSplit(file);
+}
+if (cachedEnd < start + length) {
+ length = cachedEnd - start;
+}
+if (inMemoryHosts == null) {
+ return super.makeSplit(file, start, length, hosts);
+} else {
+ return super.makeSplit(file, start, length, hosts, inMemoryHosts);
+}
+ }
+
+ private long getCachedStartIndex(Path path) throws IOException {
+if (headerCount == 0) {
+ return 0;
+}
+Long startIndexForFile = startIndexMap.get(path);
+if (startIndexForFile == null) {
+ FileSystem fileSystem;
+ FSDataInputStream fis = null;
+ fileSystem = path.getFileSystem(conf);
+ try {
+fis = fileSystem.open(path);
+for (int j = 0; j < headerCount; j++) {
+ if (fis.readLine() == null) {
+startIndexMap.put(path, Long.MAX_VALUE);
+return Long.MAX_VALUE;
+ }
+}
+// back 1 byte because readers skip the entire first row if split
start is not 0
+startIndexForFile = fis.getPos() - 1;
+ } finally {
+if (fis != null) {
+ fis.close();
+}
+ }
+ startIndexMap.put(path, startIndexForFile);
+}
+return startIndexForFile;
+ }
+
[jira] [Commented] (HIVE-22270) Upgrade commons-io to 2.6
[
https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944161#comment-16944161
]
Hive QA commented on HIVE-22270:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18859/dev-support/hive-personality.sh
|
| git revision | master / 8bd0e2a |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18859/yetus/patch-asflicense-problems.txt
|
| modules | C: . testutils/ptest2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18859/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade commons-io to 2.6
> -
>
> Key: HIVE-22270
> URL: https://issues.apache.org/jira/browse/HIVE-22270
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22270.01.patch, HIVE-22270.01.patch,
> HIVE-22270.patch, HIVE-22270.patch, HIVE-22270.patch, HIVE-22270.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using commons-io 2.4 and according to HIVE-21273, a number
> of issues are present in it, which can be resolved by upgrading to 2.6:
> IOUtils copyLarge() and skip() methods are performance hogs
> affectsVersions:2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues]
> CharSequenceInputStream#reset() behaves incorrectly in case when buffer size
> is not dividable by data size
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues]
> [Tailer] InterruptedException while the thead is sleeping is silently ignored
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues]
> IOUtils.contentEquals* methods returns false if input1 == input2; should
> return true
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues]
> Apache Commons - standard links for documents are failing
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues]
> FileUtils.sizeOfDirectoryAsBigInteger can overflow
> affectsVersions:2.4
>
>
[jira] [Commented] (HIVE-22212) Implement append partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944153#comment-16944153
]
Hive QA commented on HIVE-22212:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982110/HIVE-22212.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17234 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18858/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18858/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18858/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982110 - PreCommit-HIVE-Build
> Implement append partition related methods on temporary tables
> --
>
> Key: HIVE-22212
> URL: https://issues.apache.org/jira/browse/HIVE-22212
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22212.01.patch, HIVE-22212.02.patch,
> HIVE-22212.03.patch
>
>
> The following methods must be implemented in SessionHiveMetastoreClient, in
> order to support partition append on temporary tables:
> {code:java}
> Partition appendPartition(String dbName, String tableName, List
> partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> List partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String dbName, String tableName, String name)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> String name)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-20150) TopNKey pushdown
[ https://issues.apache.org/jira/browse/HIVE-20150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang reassigned HIVE-20150: Assignee: Richard Zhang (was: Krisztian Kasa) > TopNKey pushdown > > > Key: HIVE-20150 > URL: https://issues.apache.org/jira/browse/HIVE-20150 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Teddy Choi >Assignee: Richard Zhang >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20150.1.patch, HIVE-20150.10.patch, > HIVE-20150.11.patch, HIVE-20150.11.patch, HIVE-20150.2.patch, > HIVE-20150.4.patch, HIVE-20150.5.patch, HIVE-20150.6.patch, > HIVE-20150.7.patch, HIVE-20150.8.patch, HIVE-20150.9.patch > > > TopNKey operator is implemented in HIVE-17896, but it needs more work in > pushdown implementation. So this issue covers TopNKey pushdown implementation > with proper tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22212) Implement append partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944137#comment-16944137
]
Hive QA commented on HIVE-22212:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18858/dev-support/hive-personality.sh
|
| git revision | master / 8bd0e2a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18858/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement append partition related methods on temporary tables
> --
>
> Key: HIVE-22212
> URL: https://issues.apache.org/jira/browse/HIVE-22212
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22212.01.patch, HIVE-22212.02.patch,
> HIVE-22212.03.patch
>
>
> The following methods must be implemented in SessionHiveMetastoreClient, in
> order to support partition append on temporary tables:
> {code:java}
> Partition appendPartition(String dbName, String tableName, List
> partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> List partVals)
> throws
[jira] [Commented] (HIVE-22248) Min value for column in stats is not set correctly for some data types
[
https://issues.apache.org/jira/browse/HIVE-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944119#comment-16944119
]
Hive QA commented on HIVE-22248:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982108/HIVE-22248.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 17138 tests
executed
*Failed tests:*
{noformat}
TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=233)
TestObjectStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=233)
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultClient.getPartitions
(batchId=223)
org.apache.hadoop.hive.ql.TestTxnCommands.testSimpleAcidInsert (batchId=353)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18857/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18857/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18857/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982108 - PreCommit-HIVE-Build
> Min value for column in stats is not set correctly for some data types
> --
>
> Key: HIVE-22248
> URL: https://issues.apache.org/jira/browse/HIVE-22248
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22248.01.patch, HIVE-22248.03.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I am not sure whether the problem is printing the value or in the value
> stored in the metastore itself, but for some types (e.g. tinyint, smallint,
> int, bigint, double or float), the min value does not seem to be set
> correctly (set to 0).
> https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision
[
https://issues.apache.org/jira/browse/HIVE-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-14302:
--
Labels: pull-request-available (was: )
> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-14302.patch
>
>
> Decimal support in the optimized hashtable was decided on the basis of the
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
> TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
> predicate: (a is not null and true) (type: boolean)
> Select Operator (SEL_2)
> expressions: a (type: decimal(10,2))
> outputColumnNames: _col0
> Reduce Output Operator (RS_6)
> key expressions: _col0 (type: decimal(11,2))
> sort order: +
> Map-reduce partition columns: _col0 (type: decimal(11,2))
> Join Operator (JOIN_8)
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col0 (type: decimal(11,2))
> 1 _col0 (type: decimal(11,2))
> Group By Operator (GBY_11)
> aggregations: count(1)
> mode: hash
> outputColumnNames: _col0
> {code}
> See cast up to Decimal(11, 2) in the plan, which normalizes both sides of the
> join to be able to compare HiveDecimal as-is.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision
[
https://issues.apache.org/jira/browse/HIVE-14302?focusedWorklogId=323040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-323040
]
ASF GitHub Bot logged work on HIVE-14302:
-
Author: ASF GitHub Bot
Created on: 03/Oct/19 23:16
Start Date: 03/Oct/19 23:16
Worklog Time Spent: 10m
Work Description: mustafaiman commented on pull request #803: HIVE-14302
URL: https://github.com/apache/hive/pull/803
Adds decimal to MapJoinKey supported fields.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 323040)
Remaining Estimate: 0h
Time Spent: 10m
> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-14302.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Decimal support in the optimized hashtable was decided on the basis of the
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
> TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
> predicate: (a is not null and true) (type: boolean)
> Select Operator (SEL_2)
> expressions: a (type: decimal(10,2))
> outputColumnNames: _col0
> Reduce Output Operator (RS_6)
> key expressions: _col0 (type: decimal(11,2))
> sort order: +
> Map-reduce partition columns: _col0 (type: decimal(11,2))
> Join Operator (JOIN_8)
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col0 (type: decimal(11,2))
> 1 _col0 (type: decimal(11,2))
> Group By Operator (GBY_11)
> aggregations: count(1)
> mode: hash
> outputColumnNames: _col0
> {code}
> See cast up to Decimal(11, 2) in the plan, which normalizes both sides of the
> join to be able to compare HiveDecimal as-is.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision
[
https://issues.apache.org/jira/browse/HIVE-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mustafa Iman reassigned HIVE-14302:
---
Assignee: Mustafa Iman (was: Gopal Vijayaraghavan)
> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-14302.patch
>
>
> Decimal support in the optimized hashtable was decided on the basis of the
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
> TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
> predicate: (a is not null and true) (type: boolean)
> Select Operator (SEL_2)
> expressions: a (type: decimal(10,2))
> outputColumnNames: _col0
> Reduce Output Operator (RS_6)
> key expressions: _col0 (type: decimal(11,2))
> sort order: +
> Map-reduce partition columns: _col0 (type: decimal(11,2))
> Join Operator (JOIN_8)
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col0 (type: decimal(11,2))
> 1 _col0 (type: decimal(11,2))
> Group By Operator (GBY_11)
> aggregations: count(1)
> mode: hash
> outputColumnNames: _col0
> {code}
> See cast up to Decimal(11, 2) in the plan, which normalizes both sides of the
> join to be able to compare HiveDecimal as-is.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-14302) Tez: Optimized Hashtable can support DECIMAL keys of same precision
[
https://issues.apache.org/jira/browse/HIVE-14302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mustafa Iman updated HIVE-14302:
Attachment: HIVE-14302.patch
Status: Patch Available (was: Open)
> Tez: Optimized Hashtable can support DECIMAL keys of same precision
> ---
>
> Key: HIVE-14302
> URL: https://issues.apache.org/jira/browse/HIVE-14302
> Project: Hive
> Issue Type: Improvement
> Components: Tez
>Affects Versions: 2.2.0
>Reporter: Gopal Vijayaraghavan
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-14302.patch
>
>
> Decimal support in the optimized hashtable was decided on the basis of the
> fact that Decimal(10,1) == Decimal(10, 2) when both contain "1.0" and "1.00".
> However, the joins now don't have any issues with decimal precision because
> they cast to common.
> {code}
> create temporary table x (a decimal(10,2), b decimal(10,1)) stored as orc;
> insert into x values (1.0, 1.0);
> > explain logical select count(1) from x, x x1 where x.a = x1.b;
> OK
> LOGICAL PLAN:
> $hdt$_0:$hdt$_0:x
> TableScan (TS_0)
> alias: x
> filterExpr: (a is not null and true) (type: boolean)
> Filter Operator (FIL_18)
> predicate: (a is not null and true) (type: boolean)
> Select Operator (SEL_2)
> expressions: a (type: decimal(10,2))
> outputColumnNames: _col0
> Reduce Output Operator (RS_6)
> key expressions: _col0 (type: decimal(11,2))
> sort order: +
> Map-reduce partition columns: _col0 (type: decimal(11,2))
> Join Operator (JOIN_8)
> condition map:
> Inner Join 0 to 1
> keys:
> 0 _col0 (type: decimal(11,2))
> 1 _col0 (type: decimal(11,2))
> Group By Operator (GBY_11)
> aggregations: count(1)
> mode: hash
> outputColumnNames: _col0
> {code}
> See cast up to Decimal(11, 2) in the plan, which normalizes both sides of the
> join to be able to compare HiveDecimal as-is.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22248) Min value for column in stats is not set correctly for some data types
[
https://issues.apache.org/jira/browse/HIVE-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944093#comment-16944093
]
Hive QA commented on HIVE-22248:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
9s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18857/dev-support/hive-personality.sh
|
| git revision | master / 8bd0e2a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18857/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Min value for column in stats is not set correctly for some data types
> --
>
> Key: HIVE-22248
> URL: https://issues.apache.org/jira/browse/HIVE-22248
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22248.01.patch, HIVE-22248.03.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I am not sure whether the problem is printing the value or in the value
> stored in the metastore itself, but for some types (e.g. tinyint, smallint,
> int, bigint, double or float), the min value does not seem to be set
> correctly (set to 0).
> https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944057#comment-16944057
]
Hive QA commented on HIVE-22284:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982106/HIVE-22284.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 17172 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_dp]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_struct_type_vectorization]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=159)
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testConcurrentDropPartitions
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=233)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18856/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18856/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18856/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982106 - PreCommit-HIVE-Build
> Improve LLAP CacheContentsTracker to collect and display correct statistics
> ---
>
> Key: HIVE-22284
> URL: https://issues.apache.org/jira/browse/HIVE-22284
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22284.0.patch, HIVE-22284.1.patch
>
>
> When keeping track of which buffers correspond to what Hive objects,
> CacheContentsTracker relies on cache tags.
> Currently a tag is a simple String that ideally holds DB and table name, and
> a partition spec concatenated by . and / . The information here is derived
> from the Path of the file that is getting cached. Needless to say sometimes
> this produces a wrong tag especially for external tables.
> Also there's a bug when calculating aggregated stats for a 'parent' tag
> (corresponding to the table of the partition) because the overall maxCount
> and maxSize do not add up to the sum of those in the partitions. This happens
> when buffers get removed from the cache.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944040#comment-16944040
]
Hive QA commented on HIVE-22284:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} llap-common in master has 90 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} storage-api: The patch generated 4 new + 4 unchanged -
0 fixed = 8 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 3 new + 165 unchanged - 2
fixed = 168 total (was 167) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} llap-server: The patch generated 1 new + 252 unchanged
- 13 fixed = 253 total (was 265) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
30s{color} | {color:red} storage-api generated 1 new + 48 unchanged - 0 fixed =
49 total (was 48) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:storage-api |
| | org.apache.hadoop.hive.common.io.CacheTag defines equals and uses
Object.hashCode() At CacheTag.java:Object.hashCode() At CacheTag.java:[lines
59-62] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18856/dev-support/hive-personality.sh
|
| git revision | master / 8bd0e2a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18856/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18856/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18856/yetus/diff-checkstyle-llap-server.txt
|
| findbugs |
[jira] [Commented] (HIVE-22278) Upgrade log4j to 2.12.1
[
https://issues.apache.org/jira/browse/HIVE-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944017#comment-16944017
]
Hive QA commented on HIVE-22278:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
34s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
33s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 14m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
34s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18855/dev-support/hive-personality.sh
|
| git revision | master / 212ba4b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18855/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore standalone-metastore/metastore-common ql .
itests/hive-unit testutils/ptest2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18855/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade log4j to 2.12.1
> ---
>
> Key: HIVE-22278
> URL: https://issues.apache.org/jira/browse/HIVE-22278
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22278.02.patch, HIVE-22278.02.patch,
> HIVE-22278.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using log4j 2.10.0 and
[jira] [Commented] (HIVE-22278) Upgrade log4j to 2.12.1
[
https://issues.apache.org/jira/browse/HIVE-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944000#comment-16944000
]
Hive QA commented on HIVE-22278:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982085/HIVE-22278.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=180)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar.org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar
(batchId=246)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18855/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18855/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18855/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982085 - PreCommit-HIVE-Build
> Upgrade log4j to 2.12.1
> ---
>
> Key: HIVE-22278
> URL: https://issues.apache.org/jira/browse/HIVE-22278
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22278.02.patch, HIVE-22278.02.patch,
> HIVE-22278.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using log4j 2.10.0 and according to HIVE-21273, a number of
> issues are present in it, which can be resolved by upgrading to 2.12.1:
> Curly braces in parameters are treated as placeholders
> affectsVersions:2.8.2;2.9.0;2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2032?filter=allopenissues]
> Remove Log4J API dependency on Management APIs
> affectsVersions:2.9.1;2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2126?filter=allopenissues]
> Log4j2 throws NoClassDefFoundError in Java 9
> affectsVersions:2.10.0;2.11.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2129?filter=allopenissues]
> ThreadContext map is cleared => entries are only available for one log event
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2158?filter=allopenissues]
> Objects held in SortedArrayStringMap cannot be filtered during serialization
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2163?filter=allopenissues]
> NullPointerException at
> org.apache.logging.log4j.util.Activator.loadProvider(Activator.java:81) in
> log4j 2.10.0
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2182?filter=allopenissues]
> MarkerFilter onMismatch invalid attribute in .properties
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2202?filter=allopenissues]
> Configuration builder classes should look for "onMismatch"; not "onMisMatch".
>
> affectsVersions:2.4;2.4.1;2.5;2.6;2.6.1;2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2219?filter=allopenissues]
> Empty Automatic-Module-Name Header
> affectsVersions:2.10.0;2.11.0;3.0.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2254?filter=allopenissues]
> ConcurrentModificationException from
> org.apache.logging.log4j.status.StatusLogger.(StatusLogger.java:71)
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2276?filter=allopenissues]
> Allow SystemPropertiesPropertySource to run with a SecurityManager that
> rejects system property access
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2279?filter=allopenissues]
> ParserConfigurationException when using Log4j with
> oracle.xml.jaxp.JXDocumentBuilderFactory
> affectsVersions:2.10.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2283?filter=allopenissues]
> Log4j 2.10+not working with SLF4J 1.8 in OSGI environment
> affectsVersions:2.10.0;2.11.0
>
> [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2305?filter=allopenissues]
> fix the CacheEntry map in ThrowableProxy#toExtendedStackTrace to be put and
> gotten with same key
>
[jira] [Assigned] (HIVE-20150) TopNKey pushdown
[ https://issues.apache.org/jira/browse/HIVE-20150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20150: -- Assignee: Krisztian Kasa (was: Teddy Choi) > TopNKey pushdown > > > Key: HIVE-20150 > URL: https://issues.apache.org/jira/browse/HIVE-20150 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Teddy Choi >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20150.1.patch, HIVE-20150.10.patch, > HIVE-20150.11.patch, HIVE-20150.11.patch, HIVE-20150.2.patch, > HIVE-20150.4.patch, HIVE-20150.5.patch, HIVE-20150.6.patch, > HIVE-20150.7.patch, HIVE-20150.8.patch, HIVE-20150.9.patch > > > TopNKey operator is implemented in HIVE-17896, but it needs more work in > pushdown implementation. So this issue covers TopNKey pushdown implementation > with proper tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-5: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Fix has been committed to master. Thanks for your contribution [~samuelan]. CLosing the jira. > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-5.1.patch, HIVE-5.2.patch, > HIVE-5.4.patch, Hive-5.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang reassigned HIVE-17896: Assignee: Teddy Choi (was: Richard Zhang) > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Teddy Choi >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-17896.1.patch, HIVE-17896.10.patch, > HIVE-17896.11.patch, HIVE-17896.12.patch, HIVE-17896.13.patch, > HIVE-17896.3.patch, HIVE-17896.4.patch, HIVE-17896.5.patch, > HIVE-17896.6.patch, HIVE-17896.7.patch, HIVE-17896.8.patch, HIVE-17896.9.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator
[ https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang reassigned HIVE-17896: Assignee: Richard Zhang (was: Teddy Choi) > TopNKey: Create a standalone vectorizable TopNKey operator > -- > > Key: HIVE-17896 > URL: https://issues.apache.org/jira/browse/HIVE-17896 > Project: Hive > Issue Type: New Feature > Components: Operators >Affects Versions: 3.0.0 >Reporter: Gopal Vijayaraghavan >Assignee: Richard Zhang >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-17896.1.patch, HIVE-17896.10.patch, > HIVE-17896.11.patch, HIVE-17896.12.patch, HIVE-17896.13.patch, > HIVE-17896.3.patch, HIVE-17896.4.patch, HIVE-17896.5.patch, > HIVE-17896.6.patch, HIVE-17896.7.patch, HIVE-17896.8.patch, HIVE-17896.9.patch > > > For TPC-DS Query27, the TopN operation is delayed by the group-by - the > group-by operator buffers up all the rows before discarding the 99% of the > rows in the TopN Hash within the ReduceSink Operator. > The RS TopN operator is very restrictive as it only supports doing the > filtering on the shuffle keys, but it is better to do this before breaking > the vectors into rows and losing the isRepeating properties. > Adding a TopN Key operator in the physical operator tree allows the following > to happen. > GBY->RS(Top=1) > can become > TNK(1)->GBY->RS(Top=1) > So that, the TopNKey can remove rows before they are buffered into the GBY > and consume memory. > Here's the equivalent implementation in Presto > https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35 > Adding this as a sub-feature of GroupBy prevents further optimizations if the > GBY is on keys "a,b,c" and the TopNKey is on just "a". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22282) Obtain LLAP delegation token only when LLAP is configured for Kerberos authentication
[ https://issues.apache.org/jira/browse/HIVE-22282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22282: -- Attachment: HIVE-22282.3.patch > Obtain LLAP delegation token only when LLAP is configured for Kerberos > authentication > - > > Key: HIVE-22282 > URL: https://issues.apache.org/jira/browse/HIVE-22282 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22282.1.patch, HIVE-22282.2.patch, > HIVE-22282.3.patch > > > Contains also Kerberos related Zookeeper configuration changes after refactor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22282) Obtain LLAP delegation token only when LLAP is configured for Kerberos authentication
[
https://issues.apache.org/jira/browse/HIVE-22282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943833#comment-16943833
]
Hive QA commented on HIVE-22282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982063/HIVE-22282.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 17168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=233)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=233)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18853/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18853/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18853/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982063 - PreCommit-HIVE-Build
> Obtain LLAP delegation token only when LLAP is configured for Kerberos
> authentication
> -
>
> Key: HIVE-22282
> URL: https://issues.apache.org/jira/browse/HIVE-22282
> Project: Hive
> Issue Type: Improvement
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22282.1.patch, HIVE-22282.2.patch
>
>
> Contains also Kerberos related Zookeeper configuration changes after refactor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22270) Upgrade commons-io to 2.6
[ https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-22270: Attachment: HIVE-22270.01.patch > Upgrade commons-io to 2.6 > - > > Key: HIVE-22270 > URL: https://issues.apache.org/jira/browse/HIVE-22270 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22270.01.patch, HIVE-22270.01.patch, > HIVE-22270.patch, HIVE-22270.patch, HIVE-22270.patch, HIVE-22270.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Hive's currently using commons-io 2.4 and according to HIVE-21273, a number > of issues are present in it, which can be resolved by upgrading to 2.6: > IOUtils copyLarge() and skip() methods are performance hogs > affectsVersions:2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues] > CharSequenceInputStream#reset() behaves incorrectly in case when buffer size > is not dividable by data size > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues] > [Tailer] InterruptedException while the thead is sleeping is silently ignored > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues] > IOUtils.contentEquals* methods returns false if input1 == input2; should > return true > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues] > Apache Commons - standard links for documents are failing > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues] > FileUtils.sizeOfDirectoryAsBigInteger can overflow > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues] > Regression in FileUtils.readFileToString from 2.0.1 > affectsVersions:2.1;2.2;2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-453?filter=allopenissues] > Correct exception message in FileUtils.getFile(File; String...) > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-479?filter=allopenissues] > org.apache.commons.io.FileUtils#waitFor waits too long > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-481?filter=allopenissues] > FilenameUtils should handle embedded null bytes > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-484?filter=allopenissues] > Exceptions are suppressed incorrectly when copying files. > affectsVersions:2.4;2.5 > > [https://issues.apache.org/jira/projects/IO/issues/IO-502?filter=allopenissues] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22282) Obtain LLAP delegation token only when LLAP is configured for Kerberos authentication
[
https://issues.apache.org/jira/browse/HIVE-22282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943821#comment-16943821
]
Hive QA commented on HIVE-22282:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
36s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18853/dev-support/hive-personality.sh
|
| git revision | master / 155f829 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common common llap-client
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18853/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Obtain LLAP delegation token only when LLAP is configured for Kerberos
> authentication
> -
>
> Key: HIVE-22282
> URL: https://issues.apache.org/jira/browse/HIVE-22282
> Project: Hive
> Issue Type: Improvement
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22282.1.patch, HIVE-22282.2.patch
>
>
> Contains also Kerberos related Zookeeper configuration changes after refactor.
--
This message
[jira] [Updated] (HIVE-22289) Regenerate test output for tests broken due to commit race
[ https://issues.apache.org/jira/browse/HIVE-22289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22289: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Regenerate test output for tests broken due to commit race > -- > > Key: HIVE-22289 > URL: https://issues.apache.org/jira/browse/HIVE-22289 > Project: Hive > Issue Type: Task >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22289.1.patch > > > HIVE-22042 got committed which changed the plans of a few tests (by enabling > nonstrict partitioning mode by default) then HIVE-22269 got committed which > fixes a bug with stats not being correctly calculated on some operators. Each > patch got green runs individually but together causes test output differences. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22289) Regenerate test output for tests broken due to commit race
[ https://issues.apache.org/jira/browse/HIVE-22289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943803#comment-16943803 ] Jesus Camacho Rodriguez commented on HIVE-22289: +1 I will not wait for a clean run since this is blocking all our tests. > Regenerate test output for tests broken due to commit race > -- > > Key: HIVE-22289 > URL: https://issues.apache.org/jira/browse/HIVE-22289 > Project: Hive > Issue Type: Task >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22289.1.patch > > > HIVE-22042 got committed which changed the plans of a few tests (by enabling > nonstrict partitioning mode by default) then HIVE-22269 got committed which > fixes a bug with stats not being correctly calculated on some operators. Each > patch got green runs individually but together causes test output differences. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22212) Implement append partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22212:
-
Attachment: HIVE-22212.03.patch
> Implement append partition related methods on temporary tables
> --
>
> Key: HIVE-22212
> URL: https://issues.apache.org/jira/browse/HIVE-22212
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22212.01.patch, HIVE-22212.02.patch,
> HIVE-22212.03.patch
>
>
> The following methods must be implemented in SessionHiveMetastoreClient, in
> order to support partition append on temporary tables:
> {code:java}
> Partition appendPartition(String dbName, String tableName, List
> partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> List partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String dbName, String tableName, String name)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> String name)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943790#comment-16943790
]
Hive QA commented on HIVE-22281:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982058/HIVE-22281.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18852/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18852/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18852/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12982058/HIVE-22281.2.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982058 - PreCommit-HIVE-Build
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch, HIVE-22281.2.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22212) Implement append partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943788#comment-16943788
]
Hive QA commented on HIVE-22212:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982055/HIVE-22212.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17232 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=242)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18851/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18851/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18851/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982055 - PreCommit-HIVE-Build
> Implement append partition related methods on temporary tables
> --
>
> Key: HIVE-22212
> URL: https://issues.apache.org/jira/browse/HIVE-22212
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22212.01.patch, HIVE-22212.02.patch
>
>
> The following methods must be implemented in SessionHiveMetastoreClient, in
> order to support partition append on temporary tables:
> {code:java}
> Partition appendPartition(String dbName, String tableName, List
> partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> List partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String dbName, String tableName, String name)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> String name)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22248) Min value for column in stats is not set correctly for some data types
[
https://issues.apache.org/jira/browse/HIVE-22248?focusedWorklogId=322752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322752
]
ASF GitHub Bot logged work on HIVE-22248:
-
Author: ASF GitHub Bot
Created on: 03/Oct/19 17:18
Start Date: 03/Oct/19 17:18
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #801: HIVE-22248
Fix statistics persisting issues
URL: https://github.com/apache/hive/pull/801#discussion_r331155774
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/columnstats/merge/DoubleColumnStatsMerger.java
##
@@ -30,7 +30,7 @@
public void merge(ColumnStatisticsObj aggregateColStats, ColumnStatisticsObj
newColStats) {
DoubleColumnStatsDataInspector aggregateData =
doubleInspectorFromStats(aggregateColStats);
DoubleColumnStatsDataInspector newData =
doubleInspectorFromStats(newColStats);
-aggregateData.setLowValue(Math.min(aggregateData.getLowValue(),
newData.getLowValue()));
+setMinValue(aggregateData, newData);
aggregateData.setHighValue(Math.max(aggregateData.getHighValue(),
newData.getHighValue()));
Review comment:
Should we do the same for high value?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322752)
Time Spent: 40m (was: 0.5h)
> Min value for column in stats is not set correctly for some data types
> --
>
> Key: HIVE-22248
> URL: https://issues.apache.org/jira/browse/HIVE-22248
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22248.01.patch, HIVE-22248.03.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I am not sure whether the problem is printing the value or in the value
> stored in the metastore itself, but for some types (e.g. tinyint, smallint,
> int, bigint, double or float), the min value does not seem to be set
> correctly (set to 0).
> https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22290) ObjectStore.cleanWriteNotificationEvents and ObjectStore.cleanupEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naresh P R reassigned HIVE-22290: - Assignee: Naresh P R > ObjectStore.cleanWriteNotificationEvents and ObjectStore.cleanupEvents > OutOfMemory on large number of pending events > > > Key: HIVE-22290 > URL: https://issues.apache.org/jira/browse/HIVE-22290 > Project: Hive > Issue Type: Bug > Components: HCatalog, repl >Affects Versions: 4.0.0 >Reporter: Thomas Prelle >Assignee: Naresh P R >Priority: Major > > As in [https://jira.apache.org/jira/browse/HIVE-19430] if there are large > number of events that haven't been cleaned up for some reason, then > ObjectStore.cleanWriteNotificationEvents() and ObjectStore.cleanupEvents can > run out of memory while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22212) Implement append partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943753#comment-16943753
]
Hive QA commented on HIVE-22212:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18851/dev-support/hive-personality.sh
|
| git revision | master / 155f829 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18851/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement append partition related methods on temporary tables
> --
>
> Key: HIVE-22212
> URL: https://issues.apache.org/jira/browse/HIVE-22212
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22212.01.patch, HIVE-22212.02.patch
>
>
> The following methods must be implemented in SessionHiveMetastoreClient, in
> order to support partition append on temporary tables:
> {code:java}
> Partition appendPartition(String dbName, String tableName, List
> partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> List partVals)
> throws InvalidObjectException,
[jira] [Updated] (HIVE-22248) Min value for column in stats is not set correctly for some data types
[ https://issues.apache.org/jira/browse/HIVE-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22248: -- Attachment: HIVE-22248.03.patch > Min value for column in stats is not set correctly for some data types > -- > > Key: HIVE-22248 > URL: https://issues.apache.org/jira/browse/HIVE-22248 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22248.01.patch, HIVE-22248.03.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > I am not sure whether the problem is printing the value or in the value > stored in the metastore itself, but for some types (e.g. tinyint, smallint, > int, bigint, double or float), the min value does not seem to be set > correctly (set to 0). > https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22248) Min value for column in stats is not set correctly for some data types
[ https://issues.apache.org/jira/browse/HIVE-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22248: -- Attachment: (was: HIVE-22248.02.patch) > Min value for column in stats is not set correctly for some data types > -- > > Key: HIVE-22248 > URL: https://issues.apache.org/jira/browse/HIVE-22248 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22248.01.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > I am not sure whether the problem is printing the value or in the value > stored in the metastore itself, but for some types (e.g. tinyint, smallint, > int, bigint, double or float), the min value does not seem to be set > correctly (set to 0). > https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21025) LLAP IO fails on read if partition column is included in the table and the query has a predicate on the partition column
[
https://issues.apache.org/jira/browse/HIVE-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943705#comment-16943705
]
Mustafa Iman commented on HIVE-21025:
-
Is there an example of such cases? I tried creating text, orc and avro external
tables having partitioning column in the data. Hive overlooks such data and
runs without any errors in my tests.
> LLAP IO fails on read if partition column is included in the table and the
> query has a predicate on the partition column
>
>
> Key: HIVE-21025
> URL: https://issues.apache.org/jira/browse/HIVE-21025
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.3.4
>Reporter: Eugene Koifman
>Assignee: Mustafa Iman
>Priority: Major
>
> Hive doesn't officially support the case when a partitioning column is also
> included in the data itself, though it works in some cases. Hive would never
> write a data file with partition column in it but this can happen for
> external tables where data is added by the end user.
> Consider improving validation (at least for schema-aware files) on read to
> produce a better error than {{ArrayIndexOutOfBoundsException}}
> {code:java}
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> ], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) :
> attempt_1539023000868_24675_3_01_07_3:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException: java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:218)
>
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
>
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
>
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
>
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
>
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
>
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:80)
>
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
>
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:189)
>
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
>
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
>
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
>
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
>
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
>
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
>
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
>
> at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
>
> ... 17 more
> Caused by: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.hadoop.hive.llap.io.api.impl.LlapRecordReader.rethrowErrorIfAny(LlapRecordReader.java:355)
>
> at
>
[jira] [Commented] (HIVE-22270) Upgrade commons-io to 2.6
[
https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943701#comment-16943701
]
Hive QA commented on HIVE-22270:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982051/HIVE-22270.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=112)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18850/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18850/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18850/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982051 - PreCommit-HIVE-Build
> Upgrade commons-io to 2.6
> -
>
> Key: HIVE-22270
> URL: https://issues.apache.org/jira/browse/HIVE-22270
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22270.01.patch, HIVE-22270.patch, HIVE-22270.patch,
> HIVE-22270.patch, HIVE-22270.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using commons-io 2.4 and according to HIVE-21273, a number
> of issues are present in it, which can be resolved by upgrading to 2.6:
> IOUtils copyLarge() and skip() methods are performance hogs
> affectsVersions:2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues]
> CharSequenceInputStream#reset() behaves incorrectly in case when buffer size
> is not dividable by data size
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues]
> [Tailer] InterruptedException while the thead is sleeping is silently ignored
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues]
> IOUtils.contentEquals* methods returns false if input1 == input2; should
> return true
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues]
> Apache Commons - standard links for documents are failing
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues]
> FileUtils.sizeOfDirectoryAsBigInteger can overflow
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues]
> Regression in FileUtils.readFileToString from 2.0.1
> affectsVersions:2.1;2.2;2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-453?filter=allopenissues]
> Correct exception message in FileUtils.getFile(File; String...)
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-479?filter=allopenissues]
> org.apache.commons.io.FileUtils#waitFor waits too long
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-481?filter=allopenissues]
> FilenameUtils should handle embedded null bytes
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-484?filter=allopenissues]
> Exceptions are suppressed incorrectly when copying files.
> affectsVersions:2.4;2.5
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-502?filter=allopenissues]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22270) Upgrade commons-io to 2.6
[
https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943682#comment-16943682
]
Hive QA commented on HIVE-22270:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18850/dev-support/hive-personality.sh
|
| git revision | master / 155f829 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18850/yetus/patch-asflicense-problems.txt
|
| modules | C: . testutils/ptest2 U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18850/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade commons-io to 2.6
> -
>
> Key: HIVE-22270
> URL: https://issues.apache.org/jira/browse/HIVE-22270
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22270.01.patch, HIVE-22270.patch, HIVE-22270.patch,
> HIVE-22270.patch, HIVE-22270.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using commons-io 2.4 and according to HIVE-21273, a number
> of issues are present in it, which can be resolved by upgrading to 2.6:
> IOUtils copyLarge() and skip() methods are performance hogs
> affectsVersions:2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues]
> CharSequenceInputStream#reset() behaves incorrectly in case when buffer size
> is not dividable by data size
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues]
> [Tailer] InterruptedException while the thead is sleeping is silently ignored
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues]
> IOUtils.contentEquals* methods returns false if input1 == input2; should
> return true
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues]
> Apache Commons - standard links for documents are failing
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues]
> FileUtils.sizeOfDirectoryAsBigInteger can overflow
> affectsVersions:2.4
>
>
[jira] [Updated] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[ https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22284: -- Status: Patch Available (was: In Progress) > Improve LLAP CacheContentsTracker to collect and display correct statistics > --- > > Key: HIVE-22284 > URL: https://issues.apache.org/jira/browse/HIVE-22284 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22284.0.patch, HIVE-22284.1.patch > > > When keeping track of which buffers correspond to what Hive objects, > CacheContentsTracker relies on cache tags. > Currently a tag is a simple String that ideally holds DB and table name, and > a partition spec concatenated by . and / . The information here is derived > from the Path of the file that is getting cached. Needless to say sometimes > this produces a wrong tag especially for external tables. > Also there's a bug when calculating aggregated stats for a 'parent' tag > (corresponding to the table of the partition) because the overall maxCount > and maxSize do not add up to the sum of those in the partitions. This happens > when buffers get removed from the cache. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[ https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22284: -- Status: In Progress (was: Patch Available) > Improve LLAP CacheContentsTracker to collect and display correct statistics > --- > > Key: HIVE-22284 > URL: https://issues.apache.org/jira/browse/HIVE-22284 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22284.0.patch, HIVE-22284.1.patch > > > When keeping track of which buffers correspond to what Hive objects, > CacheContentsTracker relies on cache tags. > Currently a tag is a simple String that ideally holds DB and table name, and > a partition spec concatenated by . and / . The information here is derived > from the Path of the file that is getting cached. Needless to say sometimes > this produces a wrong tag especially for external tables. > Also there's a bug when calculating aggregated stats for a 'parent' tag > (corresponding to the table of the partition) because the overall maxCount > and maxSize do not add up to the sum of those in the partitions. This happens > when buffers get removed from the cache. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[ https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22284: -- Attachment: HIVE-22284.1.patch > Improve LLAP CacheContentsTracker to collect and display correct statistics > --- > > Key: HIVE-22284 > URL: https://issues.apache.org/jira/browse/HIVE-22284 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22284.0.patch, HIVE-22284.1.patch > > > When keeping track of which buffers correspond to what Hive objects, > CacheContentsTracker relies on cache tags. > Currently a tag is a simple String that ideally holds DB and table name, and > a partition spec concatenated by . and / . The information here is derived > from the Path of the file that is getting cached. Needless to say sometimes > this produces a wrong tag especially for external tables. > Also there's a bug when calculating aggregated stats for a 'parent' tag > (corresponding to the table of the partition) because the overall maxCount > and maxSize do not add up to the sum of those in the partitions. This happens > when buffers get removed from the cache. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22267) Support password based authentication in HMS
[
https://issues.apache.org/jira/browse/HIVE-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943642#comment-16943642
]
Hive QA commented on HIVE-22267:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982054/HIVE-22267.08.patch
{color:green}SUCCESS:{color} +1 due to 23 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17448 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18849/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18849/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18849/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982054 - PreCommit-HIVE-Build
> Support password based authentication in HMS
>
>
> Key: HIVE-22267
> URL: https://issues.apache.org/jira/browse/HIVE-22267
> Project: Hive
> Issue Type: New Feature
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22267.00.patch, HIVE-22267.01.patch,
> HIVE-22267.02.patch, HIVE-22267.03.patch, HIVE-22267.04.patch,
> HIVE-22267.05.patch, HIVE-22267.06.patch, HIVE-22267.07.patch,
> HIVE-22267.08.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Similar to HS2, support password based authentication in HMS.
> Right now we provide LDAP and CONFIG based options. The later allows to set
> user and password in config and is used only for testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22249) Support Parquet through HCatalog
[ https://issues.apache.org/jira/browse/HIVE-22249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943630#comment-16943630 ] Mass Dosage commented on HIVE-22249: Hey [~pvary] - just a gentle reminder to commit if you have the time. Thanks! > Support Parquet through HCatalog > > > Key: HIVE-22249 > URL: https://issues.apache.org/jira/browse/HIVE-22249 > Project: Hive > Issue Type: New Feature >Reporter: Jay Green-Stevens >Assignee: Jay Green-Stevens >Priority: Major > Fix For: 2.3.6 > > Attachments: HIVE-22249.2.branch-2.3.patch, > HIVE-22249.branch-2.3.patch > > > HIVE-8838 added Parquet support to HCatalog for Hive 3.0.0. We would like to > backport this functionality to Hive 2.x (primarily 2.3.x) for users who are > currently unable to migrate to Hive 3. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22267) Support password based authentication in HMS
[
https://issues.apache.org/jira/browse/HIVE-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943627#comment-16943627
]
Hive QA commented on HIVE-22267:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
39s{color} | {color:blue} standalone-metastore/metastore-common in master has
32 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
170 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} standalone-metastore: The patch generated 142 new +
949 unchanged - 3 fixed = 1091 total (was 952) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 34 new + 324 unchanged - 0 fixed = 358 total (was 324) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 108 new + 625 unchanged - 3 fixed = 733 total (was 628) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} itests/hive-minikdc: The patch generated 6 new + 19
unchanged - 0 fixed = 25 total (was 19) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} itests/hive-unit-hadoop2: The patch generated 1 new +
17 unchanged - 0 fixed = 18 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
46s{color} | {color:red} standalone-metastore/metastore-common generated 5 new
+ 32 unchanged - 0 fixed = 37 total (was 32) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
16s{color} | {color:red} standalone-metastore generated 15 new + 77 unchanged -
0 fixed = 92 total (was 77) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
57s{color} | {color:red} standalone-metastore_metastore-common generated 15 new
+ 51 unchanged - 0 fixed = 66 total (was 51) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| | The method name
org.apache.hadoop.hive.metastore.MetaStoreAnonymousAuthenticationProviderImpl.Authenticate(String,
String) doesn't start with a lower case letter At
MetaStoreAnonymousAuthenticationProviderImpl.java:doesn't start with a lower
case letter At MetaStoreAnonymousAuthenticationProviderImpl.java:[line 31] |
| | The method name
[jira] [Commented] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943601#comment-16943601
]
Hive QA commented on HIVE-22281:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982058/HIVE-22281.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17169 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18848/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18848/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18848/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982058 - PreCommit-HIVE-Build
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch, HIVE-22281.2.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21146) Enforce TransactionBatch size=1 for blob stores
[
https://issues.apache.org/jira/browse/HIVE-21146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943578#comment-16943578
]
David Lavati commented on HIVE-21146:
-
All test failures are due to this new validation step, it seems like the scope
of affected filesystems has to be limited. I've yet to find a proper solution
for this, as e.g. AzureBlobFileSystem and its store are simply extending their
base classes.
> Enforce TransactionBatch size=1 for blob stores
> ---
>
> Key: HIVE-21146
> URL: https://issues.apache.org/jira/browse/HIVE-21146
> Project: Hive
> Issue Type: Bug
> Components: Streaming, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21146.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Streaming Ingest API supports a concept of {{TransactionBatch}} where N
> transactions can be opened at once and the data in all of them will be
> written to the same delta_x_y directory where each transaction in the batch
> can be committed/aborted independently. The implementation relies on
> {{FSDataOutputStream.hflush()}} (called from OrcRecordUpdater}} which is
> available on HDFS but is often implemented as no-op in Blob store backed
> {{FileSystem}} objects.
> Need to add a check to {{HiveStreamingConnection()}} constructor to raise an
> error if {{builder.transactionBatchSize > 1}} and the target table/partitions
> are backed by something that doesn't support {{hflush()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943575#comment-16943575
]
Hive QA commented on HIVE-22281:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
37s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18848/dev-support/hive-personality.sh
|
| git revision | master / 155f829 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18848/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch, HIVE-22281.2.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22250) Describe function does not provide description for rank functions
[
https://issues.apache.org/jira/browse/HIVE-22250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943552#comment-16943552
]
Hive QA commented on HIVE-22250:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982045/HIVE-22250.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 17168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_percentile_cont]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_percentile_disc]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bigint] (batchId=93)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_boolean] (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_double] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_float] (batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_int] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_smallint]
(batchId=99)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_tinyint] (batchId=17)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18847/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18847/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18847/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982045 - PreCommit-HIVE-Build
> Describe function does not provide description for rank functions
> -
>
> Key: HIVE-22250
> URL: https://issues.apache.org/jira/browse/HIVE-22250
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22250.1.patch, HIVE-22250.1.patch,
> HIVE-22250.1.patch, HIVE-22250.2.patch, HIVE-22250.3.patch
>
>
> {code}
> @WindowFunctionDescription(
> description = @Description(
> name = "dense_rank",
> value = "_FUNC_(x) The difference between RANK and DENSE_RANK is that
> DENSE_RANK leaves no " +
> "gaps in ranking sequence when there are ties. That is, if you
> were " +
> "ranking a competition using DENSE_RANK and had three people tie
> for " +
> "second place, you would say that all three were in second place
> and " +
> "that the next person came in third."
> ),
> supportsWindow = false,
> pivotResult = true,
> rankingFunction = true,
> impliesOrder = true
> )
> {code}
> {code}
> DESC FUNCTION dense_rank;
> {code}
> {code}
> PREHOOK: query: DESC FUNCTION dense_rank
> PREHOOK: type: DESCFUNCTION
> POSTHOOK: query: DESC FUNCTION dense_rank
> POSTHOOK: type: DESCFUNCTION
> There is no documentation for function 'dense_rank'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22278) Upgrade log4j to 2.12.1
[ https://issues.apache.org/jira/browse/HIVE-22278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-22278: Attachment: HIVE-22278.02.patch > Upgrade log4j to 2.12.1 > --- > > Key: HIVE-22278 > URL: https://issues.apache.org/jira/browse/HIVE-22278 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22278.02.patch, HIVE-22278.02.patch, > HIVE-22278.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Hive's currently using log4j 2.10.0 and according to HIVE-21273, a number of > issues are present in it, which can be resolved by upgrading to 2.12.1: > Curly braces in parameters are treated as placeholders > affectsVersions:2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2032?filter=allopenissues] > Remove Log4J API dependency on Management APIs > affectsVersions:2.9.1;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2126?filter=allopenissues] > Log4j2 throws NoClassDefFoundError in Java 9 > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2129?filter=allopenissues] > ThreadContext map is cleared => entries are only available for one log event > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2158?filter=allopenissues] > Objects held in SortedArrayStringMap cannot be filtered during serialization > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2163?filter=allopenissues] > NullPointerException at > org.apache.logging.log4j.util.Activator.loadProvider(Activator.java:81) in > log4j 2.10.0 > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2182?filter=allopenissues] > MarkerFilter onMismatch invalid attribute in .properties > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2202?filter=allopenissues] > Configuration builder classes should look for "onMismatch"; not "onMisMatch". > > affectsVersions:2.4;2.4.1;2.5;2.6;2.6.1;2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2219?filter=allopenissues] > Empty Automatic-Module-Name Header > affectsVersions:2.10.0;2.11.0;3.0.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2254?filter=allopenissues] > ConcurrentModificationException from > org.apache.logging.log4j.status.StatusLogger.(StatusLogger.java:71) > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2276?filter=allopenissues] > Allow SystemPropertiesPropertySource to run with a SecurityManager that > rejects system property access > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2279?filter=allopenissues] > ParserConfigurationException when using Log4j with > oracle.xml.jaxp.JXDocumentBuilderFactory > affectsVersions:2.10.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2283?filter=allopenissues] > Log4j 2.10+not working with SLF4J 1.8 in OSGI environment > affectsVersions:2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2305?filter=allopenissues] > fix the CacheEntry map in ThrowableProxy#toExtendedStackTrace to be put and > gotten with same key > affectsVersions:2.6.2;2.7;2.8;2.8.1;2.8.2;2.9.0;2.9.1;2.10.0;2.11.0 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2389?filter=allopenissues] > NullPointerException when closing never used RollingRandomAccessFileAppender > affectsVersions:2.10.0;2.11.1 > > [https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2418?filter=allopenissues] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22250) Describe function does not provide description for rank functions
[
https://issues.apache.org/jira/browse/HIVE-22250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943525#comment-16943525
]
Hive QA commented on HIVE-22250:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 26 new + 289 unchanged - 23
fixed = 315 total (was 312) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18847/dev-support/hive-personality.sh
|
| git revision | master / 155f829 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18847/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18847/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Describe function does not provide description for rank functions
> -
>
> Key: HIVE-22250
> URL: https://issues.apache.org/jira/browse/HIVE-22250
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22250.1.patch, HIVE-22250.1.patch,
> HIVE-22250.1.patch, HIVE-22250.2.patch, HIVE-22250.3.patch
>
>
> {code}
> @WindowFunctionDescription(
> description = @Description(
> name = "dense_rank",
> value = "_FUNC_(x) The difference between RANK and DENSE_RANK is that
> DENSE_RANK leaves no " +
> "gaps in ranking sequence when there are ties. That is, if you
> were " +
> "ranking a competition using DENSE_RANK and had three people tie
> for " +
> "second place, you would say that all three were in second place
> and " +
> "that the next person came in third."
> ),
> supportsWindow = false,
> pivotResult = true,
> rankingFunction = true,
> impliesOrder = true
> )
> {code}
> {code}
> DESC FUNCTION dense_rank;
> {code}
> {code}
> PREHOOK: query: DESC FUNCTION dense_rank
> PREHOOK: type: DESCFUNCTION
> POSTHOOK: query: DESC FUNCTION dense_rank
> POSTHOOK: type: DESCFUNCTION
> There is no documentation for function 'dense_rank'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22289) Regenerate test output for tests broken due to commit race
[ https://issues.apache.org/jira/browse/HIVE-22289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-22289: Attachment: HIVE-22289.1.patch > Regenerate test output for tests broken due to commit race > -- > > Key: HIVE-22289 > URL: https://issues.apache.org/jira/browse/HIVE-22289 > Project: Hive > Issue Type: Task >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22289.1.patch > > > HIVE-22042 got committed which changed the plans of a few tests (by enabling > nonstrict partitioning mode by default) then HIVE-22269 got committed which > fixes a bug with stats not being correctly calculated on some operators. Each > patch got green runs individually but together causes test output differences. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22289) Regenerate test output for tests broken due to commit race
[ https://issues.apache.org/jira/browse/HIVE-22289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-22289: Status: Patch Available (was: Open) > Regenerate test output for tests broken due to commit race > -- > > Key: HIVE-22289 > URL: https://issues.apache.org/jira/browse/HIVE-22289 > Project: Hive > Issue Type: Task >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22289.1.patch > > > HIVE-22042 got committed which changed the plans of a few tests (by enabling > nonstrict partitioning mode by default) then HIVE-22269 got committed which > fixes a bug with stats not being correctly calculated on some operators. Each > patch got green runs individually but together causes test output differences. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22289) Regenerate test output for tests broken due to commit race
[ https://issues.apache.org/jira/browse/HIVE-22289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman reassigned HIVE-22289: --- > Regenerate test output for tests broken due to commit race > -- > > Key: HIVE-22289 > URL: https://issues.apache.org/jira/browse/HIVE-22289 > Project: Hive > Issue Type: Task >Reporter: John Sherman >Assignee: John Sherman >Priority: Major > > HIVE-22042 got committed which changed the plans of a few tests (by enabling > nonstrict partitioning mode by default) then HIVE-22269 got committed which > fixes a bug with stats not being correctly calculated on some operators. Each > patch got green runs individually but together causes test output differences. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22274) Upgrade Calcite version to 1.21.0
[
https://issues.apache.org/jira/browse/HIVE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943498#comment-16943498
]
Hive QA commented on HIVE-22274:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982029/HIVE-22274.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 134 failed/errored test(s), 17168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=290)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[mapjoin2]
(batchId=298)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_filter]
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_simple_select]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_simple_select]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[concat_op] (batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog3] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_partitioner]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer10]
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_union]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_case] (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_eq_with_case_when]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fold_when] (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_const_type]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_join_preds]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input9] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin2] (batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_10] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_3] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoin] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nested_column_pruning]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[optimize_filter_literal]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_nested_column_pruning]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_boolexpr]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[plan_json] (batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pointlookup] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_udf_col] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[remove_exprs_stats]
(batchId=97)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_exists]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqualcolumnrefs]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join3]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join4]
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join6]
(batchId=46)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_ppd_key_range]
(batchId=106)
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_pushdown]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime]
(batchId=198)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketpruning1]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_semijoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_semijoin]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_simple_select]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
[jira] [Commented] (HIVE-22274) Upgrade Calcite version to 1.21.0
[
https://issues.apache.org/jira/browse/HIVE-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943489#comment-16943489
]
Hive QA commented on HIVE-22274:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 7 new + 321 unchanged - 10
fixed = 328 total (was 331) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
5s{color} | {color:red} root: The patch generated 7 new + 321 unchanged - 10
fixed = 328 total (was 331) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
27s{color} | {color:red} ql generated 6 new + 1549 unchanged - 2 fixed = 1555
total (was 1551) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Dead store to joinInfo in
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveJoinFactoryImpl.createJoin(RelNode,
RelNode, RexNode, Set, JoinRelType, boolean) At
HiveRelFactories.java:org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveJoinFactoryImpl.createJoin(RelNode,
RelNode, RexNode, Set, JoinRelType, boolean) At HiveRelFactories.java:[line
161] |
| | Dead store to rightKeys in
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(LogicalCorrelate)
At
HiveRelDecorrelator.java:org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(LogicalCorrelate)
At HiveRelDecorrelator.java:[line 1465] |
| | Dead store to leftKeys in
org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(LogicalCorrelate)
At
HiveRelDecorrelator.java:org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(LogicalCorrelate)
At HiveRelDecorrelator.java:[line 1464] |
| | instanceof will always return true for all non-null values in new
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference(Join,
RexNode, RexNode), since all org.apache.calcite.rel.core.Join are instances of
org.apache.calcite.rel.core.Join At HiveRelMdPredicates.java:for all non-null
values in new
org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdPredicates$JoinConditionBasedPredicateInference(Join,
RexNode, RexNode), since all
[jira] [Work logged] (HIVE-22248) Min value for column in stats is not set correctly for some data types
[
https://issues.apache.org/jira/browse/HIVE-22248?focusedWorklogId=322490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322490
]
ASF GitHub Bot logged work on HIVE-22248:
-
Author: ASF GitHub Bot
Created on: 03/Oct/19 09:30
Start Date: 03/Oct/19 09:30
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #801:
HIVE-22248 Fix statistics persisting issues
URL: https://github.com/apache/hive/pull/801#discussion_r330944466
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/columnstats/merge/DoubleColumnStatsMerger.java
##
@@ -50,5 +50,16 @@ public void merge(ColumnStatisticsObj aggregateColStats,
ColumnStatisticsObj new
+ aggregateData.getNumDVs() + " and " + newData.getNumDVs() + " to
be " + ndv);
aggregateData.setNumDVs(ndv);
}
+
+aggregateColStats.getStatsData().setDoubleStats(aggregateData);
+ }
+
+ private double getMinValue(DoubleColumnStatsDataInspector aggregateData,
DoubleColumnStatsDataInspector newData) {
+if (!aggregateData.isSetLowValue() && !newData.isSetLowValue()) {
+ return 0;
Review comment:
My original logic was that the value for the statistics should be 0 (as we
don't have anything for that column yet), but I agree with you, it is nicer to
leave it unset in this case. In effect it's the same, but the code is cleaner.
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322490)
Time Spent: 0.5h (was: 20m)
> Min value for column in stats is not set correctly for some data types
> --
>
> Key: HIVE-22248
> URL: https://issues.apache.org/jira/browse/HIVE-22248
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22248.01.patch, HIVE-22248.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I am not sure whether the problem is printing the value or in the value
> stored in the metastore itself, but for some types (e.g. tinyint, smallint,
> int, bigint, double or float), the min value does not seem to be set
> correctly (set to 0).
> https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22282) Obtain LLAP delegation token only when LLAP is configured for Kerberos authentication
[ https://issues.apache.org/jira/browse/HIVE-22282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22282: -- Attachment: HIVE-22282.2.patch > Obtain LLAP delegation token only when LLAP is configured for Kerberos > authentication > - > > Key: HIVE-22282 > URL: https://issues.apache.org/jira/browse/HIVE-22282 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22282.1.patch, HIVE-22282.2.patch > > > Contains also Kerberos related Zookeeper configuration changes after refactor. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22136) Turn on tez.bucket.pruning
[
https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943437#comment-16943437
]
Hive QA commented on HIVE-22136:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12981821/HIVE-22136.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18844/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18844/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18844/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12981821/HIVE-22136.3.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12981821 - PreCommit-HIVE-Build
> Turn on tez.bucket.pruning
> ---
>
> Key: HIVE-22136
> URL: https://issues.apache.org/jira/browse/HIVE-22136
> Project: Hive
> Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch,
> HIVE-22136.3.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943436#comment-16943436
]
Hive QA commented on HIVE-22284:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982017/HIVE-22284.0.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18843/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18843/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18843/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982017 - PreCommit-HIVE-Build
> Improve LLAP CacheContentsTracker to collect and display correct statistics
> ---
>
> Key: HIVE-22284
> URL: https://issues.apache.org/jira/browse/HIVE-22284
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22284.0.patch
>
>
> When keeping track of which buffers correspond to what Hive objects,
> CacheContentsTracker relies on cache tags.
> Currently a tag is a simple String that ideally holds DB and table name, and
> a partition spec concatenated by . and / . The information here is derived
> from the Path of the file that is getting cached. Needless to say sometimes
> this produces a wrong tag especially for external tables.
> Also there's a bug when calculating aggregated stats for a 'parent' tag
> (corresponding to the table of the partition) because the overall maxCount
> and maxSize do not add up to the sum of those in the partitions. This happens
> when buffers get removed from the cache.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[ https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943426#comment-16943426 ] Peter Vary commented on HIVE-22284: --- [~szita]: Not an expert of this part, but one question that might be interesting? What do we expect in respect of the memory consumption for the CacheTag? Will this be a significant increase? Also maybe interning the tableName / dbName, even probably the partitionDesc strings can probably help. What do you think? > Improve LLAP CacheContentsTracker to collect and display correct statistics > --- > > Key: HIVE-22284 > URL: https://issues.apache.org/jira/browse/HIVE-22284 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22284.0.patch > > > When keeping track of which buffers correspond to what Hive objects, > CacheContentsTracker relies on cache tags. > Currently a tag is a simple String that ideally holds DB and table name, and > a partition spec concatenated by . and / . The information here is derived > from the Path of the file that is getting cached. Needless to say sometimes > this produces a wrong tag especially for external tables. > Also there's a bug when calculating aggregated stats for a 'parent' tag > (corresponding to the table of the partition) because the overall maxCount > and maxSize do not add up to the sum of those in the partitions. This happens > when buffers get removed from the cache. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22217) Better Logging for Hive JAR Reload
[ https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943418#comment-16943418 ] László Bodor commented on HIVE-22217: - [~belugabehr]: branch-3 has a minimal conflict with the master patch, could you please provide a branch-3 patch? > Better Logging for Hive JAR Reload > -- > > Key: HIVE-22217 > URL: https://issues.apache.org/jira/browse/HIVE-22217 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.2.0, 2.3.6 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22217.1.patch > > > Troubleshooting Hive Reloadable Auxiliary JARs has always been difficult. > Add logging to at least confirm which JAR files are being loaded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943411#comment-16943411
]
Hive QA commented on HIVE-22284:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} llap-common in master has 90 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} storage-api: The patch generated 4 new + 4 unchanged -
0 fixed = 8 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 6 new + 165 unchanged - 2
fixed = 171 total (was 167) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} llap-server: The patch generated 9 new + 252 unchanged
- 13 fixed = 261 total (was 265) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
31s{color} | {color:red} storage-api generated 1 new + 48 unchanged - 0 fixed =
49 total (was 48) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red} llap-server generated 2 new + 90 unchanged - 0 fixed =
92 total (was 90) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:storage-api |
| | org.apache.hadoop.hive.common.io.CacheTag defines compareTo(CacheTag) and
uses Object.equals() At CacheTag.java:Object.equals() At CacheTag.java:[lines
40-50] |
| FindBugs | module:llap-server |
| | Redundant nullcheck of state, which is known to be non-null in
org.apache.hadoop.hive.llap.cache.CacheContentsTracker.reportCached(LlapCacheableBuffer)
Redundant null check at CacheContentsTracker.java:is known to be non-null in
org.apache.hadoop.hive.llap.cache.CacheContentsTracker.reportCached(LlapCacheableBuffer)
Redundant null check at CacheContentsTracker.java:[line 94] |
| | Redundant nullcheck of state, which is known to be non-null in
org.apache.hadoop.hive.llap.cache.CacheContentsTracker.reportRemoved(LlapCacheableBuffer)
Redundant null check at CacheContentsTracker.java:is known to be non-null in
[jira] [Updated] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22281:
--
Status: Open (was: Patch Available)
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch, HIVE-22281.2.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22281:
--
Status: Patch Available (was: Open)
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch, HIVE-22281.2.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22281:
--
Attachment: HIVE-22281.2.patch
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch, HIVE-22281.2.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[
https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943394#comment-16943394
]
Hive QA commented on HIVE-22252:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982024/HIVE-22252.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=223)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18842/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18842/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18842/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982024 - PreCommit-HIVE-Build
> Fix caught NullPointerExceptions generated during EXPLAIN
> -
>
> Key: HIVE-22252
> URL: https://issues.apache.org/jira/browse/HIVE-22252
> Project: Hive
> Issue Type: Bug
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch,
> HIVE-22252.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While debugging an issue I noticed that during EXPLAIN the following methods
> throw a NullPointerException:
> VectorColumnOutputMapping#finalize
> AbstractOperatorDesc#getUserLevelStatistics
> AbstractOperatorDesc#getColumnExprMapForExplain
> The exceptions do end up getting caught but we should add null checks and
> gracefully to be less wasteful and to aid future debugging.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22212) Implement append partition related methods on temporary tables
[
https://issues.apache.org/jira/browse/HIVE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Pinter updated HIVE-22212:
-
Attachment: HIVE-22212.02.patch
> Implement append partition related methods on temporary tables
> --
>
> Key: HIVE-22212
> URL: https://issues.apache.org/jira/browse/HIVE-22212
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Laszlo Pinter
>Assignee: Laszlo Pinter
>Priority: Major
> Attachments: HIVE-22212.01.patch, HIVE-22212.02.patch
>
>
> The following methods must be implemented in SessionHiveMetastoreClient, in
> order to support partition append on temporary tables:
> {code:java}
> Partition appendPartition(String dbName, String tableName, List
> partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> List partVals)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String dbName, String tableName, String name)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> Partition appendPartition(String catName, String dbName, String tableName,
> String name)
> throws InvalidObjectException, AlreadyExistsException, MetaException,
> TException;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22267) Support password based authentication in HMS
[ https://issues.apache.org/jira/browse/HIVE-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-22267: -- Status: Open (was: Patch Available) > Support password based authentication in HMS > > > Key: HIVE-22267 > URL: https://issues.apache.org/jira/browse/HIVE-22267 > Project: Hive > Issue Type: New Feature >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22267.00.patch, HIVE-22267.01.patch, > HIVE-22267.02.patch, HIVE-22267.03.patch, HIVE-22267.04.patch, > HIVE-22267.05.patch, HIVE-22267.06.patch, HIVE-22267.07.patch, > HIVE-22267.08.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Similar to HS2, support password based authentication in HMS. > Right now we provide LDAP and CONFIG based options. The later allows to set > user and password in config and is used only for testing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22267) Support password based authentication in HMS
[ https://issues.apache.org/jira/browse/HIVE-22267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-22267: -- Attachment: HIVE-22267.08.patch Status: Patch Available (was: Open) > Support password based authentication in HMS > > > Key: HIVE-22267 > URL: https://issues.apache.org/jira/browse/HIVE-22267 > Project: Hive > Issue Type: New Feature >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22267.00.patch, HIVE-22267.01.patch, > HIVE-22267.02.patch, HIVE-22267.03.patch, HIVE-22267.04.patch, > HIVE-22267.05.patch, HIVE-22267.06.patch, HIVE-22267.07.patch, > HIVE-22267.08.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Similar to HS2, support password based authentication in HMS. > Right now we provide LDAP and CONFIG based options. The later allows to set > user and password in config and is used only for testing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22270) Upgrade commons-io to 2.6
[ https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-22270: Attachment: HIVE-22270.01.patch > Upgrade commons-io to 2.6 > - > > Key: HIVE-22270 > URL: https://issues.apache.org/jira/browse/HIVE-22270 > Project: Hive > Issue Type: Improvement >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22270.01.patch, HIVE-22270.patch, HIVE-22270.patch, > HIVE-22270.patch, HIVE-22270.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Hive's currently using commons-io 2.4 and according to HIVE-21273, a number > of issues are present in it, which can be resolved by upgrading to 2.6: > IOUtils copyLarge() and skip() methods are performance hogs > affectsVersions:2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues] > CharSequenceInputStream#reset() behaves incorrectly in case when buffer size > is not dividable by data size > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues] > [Tailer] InterruptedException while the thead is sleeping is silently ignored > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues] > IOUtils.contentEquals* methods returns false if input1 == input2; should > return true > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues] > Apache Commons - standard links for documents are failing > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues] > FileUtils.sizeOfDirectoryAsBigInteger can overflow > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues] > Regression in FileUtils.readFileToString from 2.0.1 > affectsVersions:2.1;2.2;2.3;2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-453?filter=allopenissues] > Correct exception message in FileUtils.getFile(File; String...) > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-479?filter=allopenissues] > org.apache.commons.io.FileUtils#waitFor waits too long > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-481?filter=allopenissues] > FilenameUtils should handle embedded null bytes > affectsVersions:2.4 > > [https://issues.apache.org/jira/projects/IO/issues/IO-484?filter=allopenissues] > Exceptions are suppressed incorrectly when copying files. > affectsVersions:2.4;2.5 > > [https://issues.apache.org/jira/projects/IO/issues/IO-502?filter=allopenissues] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22252) Fix caught NullPointerExceptions generated during EXPLAIN
[
https://issues.apache.org/jira/browse/HIVE-22252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943370#comment-16943370
]
Hive QA commented on HIVE-22252:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 1551 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 1
fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18842/dev-support/hive-personality.sh
|
| git revision | master / 155f829 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18842/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix caught NullPointerExceptions generated during EXPLAIN
> -
>
> Key: HIVE-22252
> URL: https://issues.apache.org/jira/browse/HIVE-22252
> Project: Hive
> Issue Type: Bug
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-22252.1.patch, HIVE-22252.2.patch,
> HIVE-22252.3.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> While debugging an issue I noticed that during EXPLAIN the following methods
> throw a NullPointerException:
> VectorColumnOutputMapping#finalize
> AbstractOperatorDesc#getUserLevelStatistics
> AbstractOperatorDesc#getColumnExprMapForExplain
> The exceptions do end up getting caught but we should add null checks and
> gracefully to be less wasteful and to aid future debugging.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22281:
--
Attachment: HIVE-22281.2.patch
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22281:
--
Status: Patch Available (was: Open)
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22281:
--
Status: Open (was: Patch Available)
> Create table statement fails with "not supported NULLS LAST for ORDER BY in
> ASC order"
> --
>
> Key: HIVE-22281
> URL: https://issues.apache.org/jira/browse/HIVE-22281
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22281.1.patch, HIVE-22281.1.patch,
> HIVE-22281.2.patch
>
>
> {code}
> CREATE TABLE table_core2c4ywq7yjx ( k1 STRING, f1 STRING,
> sequence_num BIGINT, create_bsk BIGINT, change_bsk BIGINT,
> op_code STRING ) PARTITIONED BY (run_id BIGINT) CLUSTERED BY (k1) SORTED BY
> (k1, change_bsk, sequence_num) INTO 4 BUCKETS STORED AS ORC
> {code}
> {code}
> Error while compiling statement: FAILED: SemanticException create/alter
> table: not supported NULLS LAST for ORDER BY in ASC order
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22281) Create table statement fails with "not supported NULLS LAST for ORDER BY in ASC order"
[
https://issues.apache.org/jira/browse/HIVE-22281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943355#comment-16943355
]
Hive QA commented on HIVE-22281:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982025/HIVE-22281.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18841/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18841/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18841/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-10-03 06:11:57.671
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-18841/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-10-03 06:11:57.674
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 155f829 HIVE-22207: Tez SplitGenerator throws
NumberFormatException when "dfs.blocksize" on cluster is "128m" (Naresh P R,
reviewed by Sankar Hariappan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 155f829 HIVE-22207: Tez SplitGenerator throws
NumberFormatException when "dfs.blocksize" on cluster is "128m" (Naresh P R,
reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-10-03 06:11:58.392
+ rm -rf ../yetus_PreCommit-HIVE-Build-18841
+ mkdir ../yetus_PreCommit-HIVE-Build-18841
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-18841
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18841/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc2077070722278764980.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc2077070722278764980.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims-0.23: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims-0.23
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-18841
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982025 - PreCommit-HIVE-Build
> Create table
[jira] [Commented] (HIVE-22270) Upgrade commons-io to 2.6
[
https://issues.apache.org/jira/browse/HIVE-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943353#comment-16943353
]
Hive QA commented on HIVE-22270:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982013/HIVE-22270.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18840/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18840/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18840/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12982013/HIVE-22270.patch was
found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982013 - PreCommit-HIVE-Build
> Upgrade commons-io to 2.6
> -
>
> Key: HIVE-22270
> URL: https://issues.apache.org/jira/browse/HIVE-22270
> Project: Hive
> Issue Type: Improvement
>Reporter: David Lavati
>Assignee: David Lavati
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22270.patch, HIVE-22270.patch, HIVE-22270.patch,
> HIVE-22270.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive's currently using commons-io 2.4 and according to HIVE-21273, a number
> of issues are present in it, which can be resolved by upgrading to 2.6:
> IOUtils copyLarge() and skip() methods are performance hogs
> affectsVersions:2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-355?filter=allopenissues]
> CharSequenceInputStream#reset() behaves incorrectly in case when buffer size
> is not dividable by data size
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-356?filter=allopenissues]
> [Tailer] InterruptedException while the thead is sleeping is silently ignored
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-357?filter=allopenissues]
> IOUtils.contentEquals* methods returns false if input1 == input2; should
> return true
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-362?filter=allopenissues]
> Apache Commons - standard links for documents are failing
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-369?filter=allopenissues]
> FileUtils.sizeOfDirectoryAsBigInteger can overflow
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-390?filter=allopenissues]
> Regression in FileUtils.readFileToString from 2.0.1
> affectsVersions:2.1;2.2;2.3;2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-453?filter=allopenissues]
> Correct exception message in FileUtils.getFile(File; String...)
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-479?filter=allopenissues]
> org.apache.commons.io.FileUtils#waitFor waits too long
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-481?filter=allopenissues]
> FilenameUtils should handle embedded null bytes
> affectsVersions:2.4
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-484?filter=allopenissues]
> Exceptions are suppressed incorrectly when copying files.
> affectsVersions:2.4;2.5
>
> [https://issues.apache.org/jira/projects/IO/issues/IO-502?filter=allopenissues]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22248) Min value for column in stats is not set correctly for some data types
[
https://issues.apache.org/jira/browse/HIVE-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943352#comment-16943352
]
Hive QA commented on HIVE-22248:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982012/HIVE-22248.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17138 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=181)
[materialized_view_create.q,reopt_dpp.q,schema_evol_orc_acid_part_update.q,orc_ppd_varchar.q,optimize_join_ptp.q,strict_managed_tables2.q,count_dist_rewrite.q,join_nullsafe.q,cross_prod_1.q,schema_evol_text_nonvec_table_llap_io.q,vectorized_shufflejoin.q,limit_pushdown.q,metadata_only_queries_with_filters.q,vector_inner_join.q,subquery_notin.q,vector_coalesce_2.q,results_cache_with_masking.q,table_access_keys_stats.q,bucket_num_reducers.q,subquery_null_agg.q,materialized_view_rewrite_6.q,keep_uniform.q,mapjoin_decimal.q,columnstats_part_coltype.q,explainanalyze_2.q,vector_char_2.q,stats_based_fetch_decision.q,sharedwork.q,column_name_is_table_alias.q,extrapolate_part_stats_partial_ndv.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=67)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge2]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_partitioned]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_ctas]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=194)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part2]
(batchId=140)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18839/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18839/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18839/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982012 - PreCommit-HIVE-Build
> Min value for column in stats is not set correctly for some data types
> --
>
> Key: HIVE-22248
> URL: https://issues.apache.org/jira/browse/HIVE-22248
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Jesus Camacho Rodriguez
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22248.01.patch, HIVE-22248.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I am not sure whether the problem is printing the value or in the value
> stored in the metastore itself, but for some types (e.g. tinyint, smallint,
> int, bigint, double or float), the min value does not seem to be set
> correctly (set to 0).
> https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/alter_table_update_status.q.out#L342
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22250) Describe function does not provide description for rank functions
[
https://issues.apache.org/jira/browse/HIVE-22250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22250:
--
Attachment: HIVE-22250.3.patch
> Describe function does not provide description for rank functions
> -
>
> Key: HIVE-22250
> URL: https://issues.apache.org/jira/browse/HIVE-22250
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22250.1.patch, HIVE-22250.1.patch,
> HIVE-22250.1.patch, HIVE-22250.2.patch, HIVE-22250.3.patch
>
>
> {code}
> @WindowFunctionDescription(
> description = @Description(
> name = "dense_rank",
> value = "_FUNC_(x) The difference between RANK and DENSE_RANK is that
> DENSE_RANK leaves no " +
> "gaps in ranking sequence when there are ties. That is, if you
> were " +
> "ranking a competition using DENSE_RANK and had three people tie
> for " +
> "second place, you would say that all three were in second place
> and " +
> "that the next person came in third."
> ),
> supportsWindow = false,
> pivotResult = true,
> rankingFunction = true,
> impliesOrder = true
> )
> {code}
> {code}
> DESC FUNCTION dense_rank;
> {code}
> {code}
> PREHOOK: query: DESC FUNCTION dense_rank
> PREHOOK: type: DESCFUNCTION
> POSTHOOK: query: DESC FUNCTION dense_rank
> POSTHOOK: type: DESCFUNCTION
> There is no documentation for function 'dense_rank'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22250) Describe function does not provide description for rank functions
[
https://issues.apache.org/jira/browse/HIVE-22250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22250:
--
Status: Patch Available (was: Open)
> Describe function does not provide description for rank functions
> -
>
> Key: HIVE-22250
> URL: https://issues.apache.org/jira/browse/HIVE-22250
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22250.1.patch, HIVE-22250.1.patch,
> HIVE-22250.1.patch, HIVE-22250.2.patch, HIVE-22250.3.patch
>
>
> {code}
> @WindowFunctionDescription(
> description = @Description(
> name = "dense_rank",
> value = "_FUNC_(x) The difference between RANK and DENSE_RANK is that
> DENSE_RANK leaves no " +
> "gaps in ranking sequence when there are ties. That is, if you
> were " +
> "ranking a competition using DENSE_RANK and had three people tie
> for " +
> "second place, you would say that all three were in second place
> and " +
> "that the next person came in third."
> ),
> supportsWindow = false,
> pivotResult = true,
> rankingFunction = true,
> impliesOrder = true
> )
> {code}
> {code}
> DESC FUNCTION dense_rank;
> {code}
> {code}
> PREHOOK: query: DESC FUNCTION dense_rank
> PREHOOK: type: DESCFUNCTION
> POSTHOOK: query: DESC FUNCTION dense_rank
> POSTHOOK: type: DESCFUNCTION
> There is no documentation for function 'dense_rank'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22250) Describe function does not provide description for rank functions
[
https://issues.apache.org/jira/browse/HIVE-22250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22250:
--
Status: Open (was: Patch Available)
> Describe function does not provide description for rank functions
> -
>
> Key: HIVE-22250
> URL: https://issues.apache.org/jira/browse/HIVE-22250
> Project: Hive
> Issue Type: Bug
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22250.1.patch, HIVE-22250.1.patch,
> HIVE-22250.1.patch, HIVE-22250.2.patch
>
>
> {code}
> @WindowFunctionDescription(
> description = @Description(
> name = "dense_rank",
> value = "_FUNC_(x) The difference between RANK and DENSE_RANK is that
> DENSE_RANK leaves no " +
> "gaps in ranking sequence when there are ties. That is, if you
> were " +
> "ranking a competition using DENSE_RANK and had three people tie
> for " +
> "second place, you would say that all three were in second place
> and " +
> "that the next person came in third."
> ),
> supportsWindow = false,
> pivotResult = true,
> rankingFunction = true,
> impliesOrder = true
> )
> {code}
> {code}
> DESC FUNCTION dense_rank;
> {code}
> {code}
> PREHOOK: query: DESC FUNCTION dense_rank
> PREHOOK: type: DESCFUNCTION
> POSTHOOK: query: DESC FUNCTION dense_rank
> POSTHOOK: type: DESCFUNCTION
> There is no documentation for function 'dense_rank'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
