Re: [PR] [WIP] HIVE-28302: Let SUM UDF return NULL when all rows have non-numeric texts [hive]
sonarcloud[bot] commented on PR #5283: URL: https://github.com/apache/hive/pull/5283#issuecomment-2152483416 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5283) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5283=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5283=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5283=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5283) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629522053
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##
@@ -275,10 +277,134 @@ private static void checkResiduals(CombinedScanTask
task) {
@Override
public RecordReader createRecordReader(InputSplit split,
TaskAttemptContext context) {
-return new IcebergRecordReader<>();
+if (split instanceof IcebergMergeSplit) {
+ return new IcebergMergeRecordReader<>();
+} else {
+ return new IcebergRecordReader<>();
+}
}
- private static final class IcebergRecordReader extends RecordReader {
+ public abstract static class AbstractIcebergRecordReader extends
RecordReader {
Review Comment:
Done.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##
@@ -436,49 +526,34 @@ private CloseableIterable openGeneric(FileScanTask
task, Schema readSchema) {
@SuppressWarnings("unchecked")
private CloseableIterable open(FileScanTask currentTask, Schema
readSchema) {
- switch (inMemoryDataModel) {
+ switch (getInMemoryDataModel()) {
Review Comment:
To avoid checkstyle errors. Checkstyle rules prevent usage of protected
variables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629521517
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##
@@ -275,10 +277,134 @@ private static void checkResiduals(CombinedScanTask
task) {
@Override
public RecordReader createRecordReader(InputSplit split,
TaskAttemptContext context) {
-return new IcebergRecordReader<>();
+if (split instanceof IcebergMergeSplit) {
Review Comment:
Done.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##
@@ -590,4 +643,135 @@ private static Schema
schemaWithoutConstantsAndMeta(Schema readSchema, Map extends
AbstractIcebergRecordReader {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629521024
##
ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java:
##
@@ -768,4 +763,8 @@ public String toString() {
public interface AvoidSplitCombination {
boolean shouldSkipCombine(Path path, Configuration conf) throws
IOException;
}
+
+ public interface MergeSplit {
Review Comment:
This is generic interface for creating merge splits and its used.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergMergeSplit.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.mapreduce;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
+import org.apache.hadoop.hive.ql.io.MergeSplitProperties;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.util.SerializationUtil;
+
+public class IcebergMergeSplit extends FileSplit implements
org.apache.hadoop.mapred.InputSplit {
Review Comment:
There are inherent problems of extending it with IcebergSplit. The required
split for merge task must be of type FileSplit however IcebergSplit extends
InputSplit only. Also the IcebergSplitContainer is also extended by
IcebergSplit which is not used here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629520820
##
ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveRecordReader.java:
##
@@ -88,14 +90,26 @@ public CombineHiveRecordReader(InputSplit split,
Configuration conf,
}
}
+FileSplit inputSplit;
// create a split for the given partition
-FileSplit fsplit = new FileSplit(hsplit.getPaths()[partition], hsplit
-.getStartOffsets()[partition], hsplit.getLengths()[partition], hsplit
-.getLocations());
+if (inputFormat instanceof CombineHiveInputFormat.MergeSplit) {
+ if (mrwork == null) {
+mrwork = (MapWork) Utilities.getMergeWork(jobConf);
+if (mrwork == null) {
Review Comment:
It can be null, hence we do Utilities.getMapWork()
##
ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java:
##
@@ -768,4 +763,8 @@ public String toString() {
public interface AvoidSplitCombination {
boolean shouldSkipCombine(Path path, Configuration conf) throws
IOException;
}
+
+ public interface MergeSplit {
Review Comment:
This is generic interface for creating merge splits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629520155
##
ql/src/java/org/apache/hadoop/hive/ql/io/MergeSplitProperties.java:
##
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.fs.Path;
+
+import java.util.Map;
+
+public interface MergeSplitProperties {
+ Map getSplitPropertyMap();
Review Comment:
Removed MergeSplitProperties. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629519719
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapred/MapredIcebergInputFormat.java:
##
@@ -85,7 +90,8 @@ private static final class MapredIcebergRecordReader
extends AbstractMapredIc
private final long splitLength; // for getPos()
MapredIcebergRecordReader(org.apache.iceberg.mr.mapreduce.IcebergInputFormat
mapreduceInputFormat,
- IcebergSplit split, JobConf job, Reporter
reporter) throws IOException {
+org.apache.hadoop.mapreduce.InputSplit split, JobConf job, Reporter
reporter)
Review Comment:
This is required since I have created a new split by the name of
IcebergMergeSplit which extends FileSplit.
##
ql/src/java/org/apache/hadoop/hive/ql/plan/MergeTaskProperties.java:
##
@@ -19,12 +19,16 @@
package org.apache.hadoop.hive.ql.plan;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.MergeSplitProperties;
import org.apache.hadoop.hive.ql.io.StorageFormatDescriptor;
import java.io.IOException;
+import java.util.Properties;
public interface MergeTaskProperties {
public Path getTmpLocation();
- public StorageFormatDescriptor getStorageFormatDescriptor() throws
IOException;
+ default MergeSplitProperties getSplitProperties() throws IOException {
+return null;
Review Comment:
Done.
##
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java:
##
@@ -184,6 +185,8 @@ public enum LlapIODescriptor {
private boolean useInputPathsDirectly;
+ private MergeSplitProperties mergeSplitProperties;
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629519259
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergMergeTaskProperties.java:
##
@@ -42,14 +46,23 @@ public Path getTmpLocation() {
return new Path(location + "/data/");
}
- public StorageFormatDescriptor getStorageFormatDescriptor() throws
IOException {
-FileFormat fileFormat =
FileFormat.fromString(properties.getProperty(TableProperties.DEFAULT_FILE_FORMAT,
-TableProperties.DEFAULT_FILE_FORMAT_DEFAULT));
-StorageFormatDescriptor descriptor =
storageFormatFactory.get(fileFormat.name());
-if (descriptor == null) {
- throw new IOException("Unsupported storage format descriptor");
+ @Override
+ public MergeSplitProperties getSplitProperties() throws IOException {
+String tableName = properties.getProperty(Catalogs.NAME);
+String snapshotRef = properties.getProperty(Catalogs.SNAPSHOT_REF);
+Configuration configuration = SessionState.getSessionConf();
+List originalContextList =
IcebergTableUtil.generateJobContext(configuration, tableName, snapshotRef);
+List jobContextList = originalContextList.stream()
+.map(TezUtil::enrichContextWithVertexId)
+.collect(Collectors.toList());
+if (jobContextList.isEmpty()) {
+ return null;
}
-return descriptor;
+List contentFiles = new
HiveIcebergOutputCommitter().getOutputContentFiles(jobContextList);
+Map pathToContentFile = Maps.newConcurrentMap();
+contentFiles.forEach(contentFile -> {
Review Comment:
I dont think it works that easily. We have to construct the path object.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java:
##
@@ -328,4 +334,40 @@ public static PartitionData toPartitionData(StructLike
key, Types.StructType key
}
return data;
}
+
+ /**
+ * Generates {@link JobContext}s for the OutputCommitter for the specific
table.
+ * @param configuration The configuration used for as a base of the JobConf
+ * @param tableName The name of the table we are planning to commit
+ * @param branchName the name of the branch
+ * @return The generated Optional JobContext list or empty if not presents.
+ */
+ static List generateJobContext(Configuration configuration,
String tableName,
Review Comment:
Done.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapred/AbstractMapredIcebergRecordReader.java:
##
@@ -23,16 +23,16 @@
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.RecordReader;
import org.apache.hadoop.mapred.Reporter;
+import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
-import org.apache.iceberg.mr.mapreduce.IcebergSplit;
@SuppressWarnings("checkstyle:VisibilityModifier")
public abstract class AbstractMapredIcebergRecordReader implements
RecordReader {
protected final org.apache.hadoop.mapreduce.RecordReader
innerReader;
public
AbstractMapredIcebergRecordReader(org.apache.iceberg.mr.mapreduce.IcebergInputFormat
mapreduceInputFormat,
- IcebergSplit split, JobConf job, Reporter reporter) throws IOException {
+ InputSplit split, JobConf job, Reporter reporter) throws IOException {
Review Comment:
This is required since I have created a new split by the name of
IcebergMergeSplit which extends FileSplit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629518521
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergMergeSplitProperties.java:
##
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.Map;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.MergeSplitProperties;
+
+public class IcebergMergeSplitProperties implements MergeSplitProperties {
Review Comment:
HashMap used. Done.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergMergeTaskProperties.java:
##
@@ -42,14 +46,23 @@ public Path getTmpLocation() {
return new Path(location + "/data/");
}
- public StorageFormatDescriptor getStorageFormatDescriptor() throws
IOException {
-FileFormat fileFormat =
FileFormat.fromString(properties.getProperty(TableProperties.DEFAULT_FILE_FORMAT,
-TableProperties.DEFAULT_FILE_FORMAT_DEFAULT));
-StorageFormatDescriptor descriptor =
storageFormatFactory.get(fileFormat.name());
-if (descriptor == null) {
- throw new IOException("Unsupported storage format descriptor");
+ @Override
+ public MergeSplitProperties getSplitProperties() throws IOException {
+String tableName = properties.getProperty(Catalogs.NAME);
+String snapshotRef = properties.getProperty(Catalogs.SNAPSHOT_REF);
+Configuration configuration = SessionState.getSessionConf();
+List originalContextList =
IcebergTableUtil.generateJobContext(configuration, tableName, snapshotRef);
Review Comment:
This is used for getting the `jobContexts` that are already present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629517880
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -136,6 +145,7 @@ public void commitTask(TaskAttemptContext originalContext)
throws IOException {
ExecutorService tableExecutor = tableExecutor(jobConf, outputs.size());
try {
// Generates commit files for the target tables in parallel
+ Collection finalMergedPaths = new
ConcurrentLinkedQueue<>(mergedPaths);
Review Comment:
1. Modified it to a list. Done.
2. This is done to ensure the information about input files used for merge
is retained. This is null in most cases except for merge.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -459,9 +474,14 @@ private void commitTable(FileIO io, ExecutorService
executor, OutputTable output
deleteFiles.addAll(writeResults.deleteFiles());
replacedDataFiles.addAll(writeResults.replacedDataFiles());
referencedDataFiles.addAll(writeResults.referencedDataFiles());
+ mergedAndDeletedFiles.addAll(writeResults.mergedAndDeletedFiles());
}
-FilesForCommit filesForCommit = new FilesForCommit(dataFiles, deleteFiles,
replacedDataFiles, referencedDataFiles);
+dataFiles.removeIf(dataFile -> mergedAndDeletedFiles.contains(new
Path(String.valueOf(dataFile.path();
Review Comment:
While writing data, there are multiple `jobContexts`. The files from one
jobContext can act as the input files for merge task which are written in
another jobContexts. Hence to resolve them, its done during commit phase.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergAcidUtil.java:
##
@@ -203,7 +204,7 @@ public static long parseFilePosition(Record rec) {
return rec.get(FILE_READ_META_COLS.get(MetadataColumns.ROW_POSITION),
Long.class);
}
- public static long computeHash(StructProjection struct) {
+ public static long computeHash(StructLike struct) {
Review Comment:
Removed. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629517567
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -783,12 +820,59 @@ public List getOutputFiles(List
jobContexts) throws IOEx
tableExecutor.shutdown();
}
}
+Collection dataFiles = new ConcurrentLinkedQueue<>();
+parentDirToDataFile.forEach((parentDir, fileStatusList) -> {
Review Comment:
Done.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -783,12 +820,59 @@ public List getOutputFiles(List
jobContexts) throws IOEx
tableExecutor.shutdown();
}
}
+Collection dataFiles = new ConcurrentLinkedQueue<>();
+parentDirToDataFile.forEach((parentDir, fileStatusList) -> {
+ if (fileStatusList.size() > 1) {
+dataFiles.addAll(fileStatusList);
+ }
+});
+parentDirToDeleteFile.forEach((parentDir, fileStatusList) -> {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629518905
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergMergeTaskProperties.java:
##
@@ -42,14 +46,23 @@ public Path getTmpLocation() {
return new Path(location + "/data/");
}
- public StorageFormatDescriptor getStorageFormatDescriptor() throws
IOException {
-FileFormat fileFormat =
FileFormat.fromString(properties.getProperty(TableProperties.DEFAULT_FILE_FORMAT,
-TableProperties.DEFAULT_FILE_FORMAT_DEFAULT));
-StorageFormatDescriptor descriptor =
storageFormatFactory.get(fileFormat.name());
-if (descriptor == null) {
- throw new IOException("Unsupported storage format descriptor");
+ @Override
+ public MergeSplitProperties getSplitProperties() throws IOException {
+String tableName = properties.getProperty(Catalogs.NAME);
+String snapshotRef = properties.getProperty(Catalogs.SNAPSHOT_REF);
+Configuration configuration = SessionState.getSessionConf();
+List originalContextList =
IcebergTableUtil.generateJobContext(configuration, tableName, snapshotRef);
+List jobContextList = originalContextList.stream()
+.map(TezUtil::enrichContextWithVertexId)
+.collect(Collectors.toList());
+if (jobContextList.isEmpty()) {
+ return null;
}
-return descriptor;
+List contentFiles = new
HiveIcebergOutputCommitter().getOutputContentFiles(jobContextList);
Review Comment:
Done. Create a static method inside HiveIcebergOutputCommitter.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergMergeTaskProperties.java:
##
@@ -42,14 +46,23 @@ public Path getTmpLocation() {
return new Path(location + "/data/");
}
- public StorageFormatDescriptor getStorageFormatDescriptor() throws
IOException {
-FileFormat fileFormat =
FileFormat.fromString(properties.getProperty(TableProperties.DEFAULT_FILE_FORMAT,
-TableProperties.DEFAULT_FILE_FORMAT_DEFAULT));
-StorageFormatDescriptor descriptor =
storageFormatFactory.get(fileFormat.name());
-if (descriptor == null) {
- throw new IOException("Unsupported storage format descriptor");
+ @Override
+ public MergeSplitProperties getSplitProperties() throws IOException {
+String tableName = properties.getProperty(Catalogs.NAME);
+String snapshotRef = properties.getProperty(Catalogs.SNAPSHOT_REF);
+Configuration configuration = SessionState.getSessionConf();
+List originalContextList =
IcebergTableUtil.generateJobContext(configuration, tableName, snapshotRef);
+List jobContextList = originalContextList.stream()
+.map(TezUtil::enrichContextWithVertexId)
+.collect(Collectors.toList());
+if (jobContextList.isEmpty()) {
+ return null;
}
-return descriptor;
+List contentFiles = new
HiveIcebergOutputCommitter().getOutputContentFiles(jobContextList);
+Map pathToContentFile = Maps.newConcurrentMap();
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629515618
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -126,6 +128,13 @@ public void commitTask(TaskAttemptContext originalContext)
throws IOException {
TaskAttemptID attemptID = context.getTaskAttemptID();
JobConf jobConf = context.getJobConf();
+Collection mergedPaths = Collections.emptySet();
Review Comment:
Done.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -459,9 +474,14 @@ private void commitTable(FileIO io, ExecutorService
executor, OutputTable output
deleteFiles.addAll(writeResults.deleteFiles());
replacedDataFiles.addAll(writeResults.replacedDataFiles());
referencedDataFiles.addAll(writeResults.referencedDataFiles());
+ mergedAndDeletedFiles.addAll(writeResults.mergedAndDeletedFiles());
}
-FilesForCommit filesForCommit = new FilesForCommit(dataFiles, deleteFiles,
replacedDataFiles, referencedDataFiles);
+dataFiles.removeIf(dataFile -> mergedAndDeletedFiles.contains(new
Path(String.valueOf(dataFile.path();
+deleteFiles.removeIf(deleteFile -> mergedAndDeletedFiles.contains(new
Path(String.valueOf(deleteFile.path();
+
+FilesForCommit filesForCommit = new FilesForCommit(dataFiles, deleteFiles,
replacedDataFiles, referencedDataFiles,
Review Comment:
So that there is nothing in the mergedAndDeletedFile list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28258: Use Iceberg semantics for Merge task [hive]
SourabhBadhya commented on code in PR #5251:
URL: https://github.com/apache/hive/pull/5251#discussion_r1629515925
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -773,8 +796,22 @@ public List getOutputFiles(List
jobContexts) throws IOEx
FilesForCommit results = collectResults(numTasks,
fileExecutor, table.location(), jobContext,
table.io(), false);
for (DataFile dataFile : results.dataFiles()) {
- FileStatus fileStatus = fileSystem.getFileStatus(new
Path(dataFile.path().toString()));
- dataFiles.add(fileStatus);
+ Path filePath = new Path(dataFile.path().toString());
+ FileStatus fileStatus = fileSystem.getFileStatus(filePath);
+ parentDirToDataFile.merge(
Review Comment:
This is to check the list of files which are part of the same parent
directory. Later this is used to decide whether they are eligible candidates
for merge.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -773,8 +796,22 @@ public List getOutputFiles(List
jobContexts) throws IOEx
FilesForCommit results = collectResults(numTasks,
fileExecutor, table.location(), jobContext,
table.io(), false);
for (DataFile dataFile : results.dataFiles()) {
- FileStatus fileStatus = fileSystem.getFileStatus(new
Path(dataFile.path().toString()));
- dataFiles.add(fileStatus);
+ Path filePath = new Path(dataFile.path().toString());
+ FileStatus fileStatus = fileSystem.getFileStatus(filePath);
+ parentDirToDataFile.merge(
+ filePath.getParent(), Lists.newArrayList(fileStatus),
(oldList, newList) -> {
+ oldList.addAll(newList);
+ return oldList;
+});
+}
+for (DeleteFile deleteFile : results.deleteFiles()) {
+ Path filePath = new Path(deleteFile.path().toString());
+ FileStatus fileStatus = fileSystem.getFileStatus(filePath);
+ parentDirToDeleteFile.merge(
Review Comment:
This is to check the list of files which are part of the same parent
directory. Later this is used to decide whether they are eligible candidates
for merge.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##
@@ -783,12 +820,59 @@ public List getOutputFiles(List
jobContexts) throws IOEx
tableExecutor.shutdown();
}
}
+Collection dataFiles = new ConcurrentLinkedQueue<>();
Review Comment:
Used List. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28304: Replace org.apache.log4j.Logger with org.slf4j.Logger [hive]
ngsg commented on PR #5280: URL: https://github.com/apache/hive/pull/5280#issuecomment-2152338917 @zhangbutao, there is no `org.apache.log4j.Logger` import in Hive project. Let me test Hive if we could remove `log4j-1.2-api` dependency or not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
abstractdog merged PR #5281: URL: https://github.com/apache/hive/pull/5281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
abstractdog commented on PR #5281: URL: https://github.com/apache/hive/pull/5281#issuecomment-2152308950 all tests passed, merging https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5281/3/pipeline -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
[PR] [WIP] HIVE-28302: Let SUM UDF return NULL when all rows have non-numeric texts [hive]
okumin opened a new pull request, #5283: URL: https://github.com/apache/hive/pull/5283 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### Is the change a dependency upgrade? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
abstractdog commented on code in PR #5281:
URL: https://github.com/apache/hive/pull/5281#discussion_r1629269032
##
common/src/java/org/apache/hive/http/ProfileServlet.java:
##
@@ -344,10 +357,11 @@ private Output getOutput(final HttpServletRequest req) {
try {
return Output.valueOf(outputArg.trim().toUpperCase());
} catch (IllegalArgumentException e) {
-return Output.SVG;
+LOG.warn("Output format value is invalid, returning with default
HTML");
Review Comment:
makes sense!
took care in
https://github.com/apache/hive/pull/5281/commits/681036c659b85b4cf1acc408dba0763137923e2b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
abstractdog commented on code in PR #5281:
URL: https://github.com/apache/hive/pull/5281#discussion_r1629269032
##
common/src/java/org/apache/hive/http/ProfileServlet.java:
##
@@ -344,10 +357,11 @@ private Output getOutput(final HttpServletRequest req) {
try {
return Output.valueOf(outputArg.trim().toUpperCase());
} catch (IllegalArgumentException e) {
-return Output.SVG;
+LOG.warn("Output format value is invalid, returning with default
HTML");
Review Comment:
makes sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
ayushtkn commented on code in PR #5281:
URL: https://github.com/apache/hive/pull/5281#discussion_r1629244225
##
common/src/java/org/apache/hive/http/ProfileServlet.java:
##
@@ -344,10 +357,11 @@ private Output getOutput(final HttpServletRequest req) {
try {
return Output.valueOf(outputArg.trim().toUpperCase());
} catch (IllegalArgumentException e) {
-return Output.SVG;
+LOG.warn("Output format value is invalid, returning with default
HTML");
Review Comment:
Can you add what value is actually configured in the log as well
```
LOG.warn("Output format value {} is invalid, returning with default
HTML", outputArg);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
sonarcloud[bot] commented on PR #5281: URL: https://github.com/apache/hive/pull/5281#issuecomment-2151916230 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5281) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5281=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5281=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5281=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5281) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
abstractdog commented on code in PR #5281:
URL: https://github.com/apache/hive/pull/5281#discussion_r1629176992
##
common/src/java/org/apache/hive/http/ProfileServlet.java:
##
@@ -293,6 +295,16 @@ protected void doGet(final HttpServletRequest req, final
HttpServletResponse res
}
}
+ /**
+ * Get the path of the profiler script to be executed.
+ * Before async-profiler 3.0, the script was named profiler.sh, and after
3.0 it's bin/asprof
+ * @return
+ */
+ private String getProfilerScriptPath() {
+return Files.exists(Paths.get(asyncProfilerHome + "/bin/asprof")) ?
(asyncProfilerHome + "/bin/asprof")
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
deniskuzZ commented on code in PR #5281:
URL: https://github.com/apache/hive/pull/5281#discussion_r1629131498
##
common/src/java/org/apache/hive/http/ProfileServlet.java:
##
@@ -141,7 +142,8 @@ enum Output {
COLLAPSED,
SVG,
Review Comment:
could we create `getProfilerVersion()` and use HTML only for newer ones?
version >= 2 ? HTML : SVG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
deniskuzZ commented on code in PR #5281:
URL: https://github.com/apache/hive/pull/5281#discussion_r1629131498
##
common/src/java/org/apache/hive/http/ProfileServlet.java:
##
@@ -141,7 +142,8 @@ enum Output {
COLLAPSED,
SVG,
Review Comment:
could we create `getProfilerVersion()` and use HTML only for newer ones?
version >= 2 ? HTML : SVG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
deniskuzZ commented on code in PR #5281:
URL: https://github.com/apache/hive/pull/5281#discussion_r1629131498
##
common/src/java/org/apache/hive/http/ProfileServlet.java:
##
@@ -141,7 +142,8 @@ enum Output {
COLLAPSED,
SVG,
Review Comment:
remove if not supported, or maybe we should create `getProfilerVersion()`
and use HTML only for newer ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
deniskuzZ commented on code in PR #5281:
URL: https://github.com/apache/hive/pull/5281#discussion_r1629131498
##
common/src/java/org/apache/hive/http/ProfileServlet.java:
##
@@ -141,7 +142,8 @@ enum Output {
COLLAPSED,
SVG,
Review Comment:
remove if not supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
deniskuzZ commented on code in PR #5281:
URL: https://github.com/apache/hive/pull/5281#discussion_r1629130338
##
common/src/java/org/apache/hive/http/ProfileServlet.java:
##
@@ -293,6 +295,16 @@ protected void doGet(final HttpServletRequest req, final
HttpServletResponse res
}
}
+ /**
+ * Get the path of the profiler script to be executed.
+ * Before async-profiler 3.0, the script was named profiler.sh, and after
3.0 it's bin/asprof
+ * @return
+ */
+ private String getProfilerScriptPath() {
+return Files.exists(Paths.get(asyncProfilerHome + "/bin/asprof")) ?
(asyncProfilerHome + "/bin/asprof")
Review Comment:
Path defaultPath = Paths.get(asyncProfilerHome + "/bin/asprof");
return Files.exists(defaultPath)? defaultPath.toString() : asyncProfilerHome
+ "/profiler.sh"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
abstractdog commented on code in PR #5281: URL: https://github.com/apache/hive/pull/5281#discussion_r1629126593 ## common/src/java/org/apache/hive/http/ProfileServlet.java: ## @@ -45,19 +47,19 @@ * // -m method fully qualified method name: 'ClassName.methodName' * // -tprofile different threads separately * // -ssimple class names instead of FQN - * // -o fmt[,fmt...] output format: summary|traces|flat|collapsed|svg|tree|jfr + * // -o fmt[,fmt...] output format: summary|traces|flat|collapsed|svg|tree|jfr|html Review Comment: it's supported in older versions of the profiler, so the user might want to use that, but the default is html now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
abstractdog commented on code in PR #5281: URL: https://github.com/apache/hive/pull/5281#discussion_r1629126593 ## common/src/java/org/apache/hive/http/ProfileServlet.java: ## @@ -45,19 +47,19 @@ * // -m method fully qualified method name: 'ClassName.methodName' * // -tprofile different threads separately * // -ssimple class names instead of FQN - * // -o fmt[,fmt...] output format: summary|traces|flat|collapsed|svg|tree|jfr + * // -o fmt[,fmt...] output format: summary|traces|flat|collapsed|svg|tree|jfr|html Review Comment: it's supported in older versions of the profiler, so the user might want to use that -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats and prepare for profiler 3.0 [hive]
deniskuzZ commented on code in PR #5281: URL: https://github.com/apache/hive/pull/5281#discussion_r1629113804 ## common/src/java/org/apache/hive/http/ProfileServlet.java: ## @@ -45,19 +47,19 @@ * // -m method fully qualified method name: 'ClassName.methodName' * // -tprofile different threads separately * // -ssimple class names instead of FQN - * // -o fmt[,fmt...] output format: summary|traces|flat|collapsed|svg|tree|jfr + * // -o fmt[,fmt...] output format: summary|traces|flat|collapsed|svg|tree|jfr|html Review Comment: should we remove `svg` if that is not supported anymore? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27498: Support custom delimiter in SkippingTextInputFormat [hive]
abstractdog merged PR #5265: URL: https://github.com/apache/hive/pull/5265 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28306: Iceberg: Return new scan after applying column project parameter [hive]
sonarcloud[bot] commented on PR #5282: URL: https://github.com/apache/hive/pull/5282#issuecomment-2151582114 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5282) **Quality Gate passed** Issues  [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive=5282=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5282=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5282=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5282) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28306: Iceberg: Return new scan after applying column project parameter [hive]
zhangbutao commented on PR #5282: URL: https://github.com/apache/hive/pull/5282#issuecomment-2151498192 Submitted a same PR in Apache Iceberg: https://github.com/apache/iceberg/pull/10449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] [WIP] Calcite upgrade 1.33 [hive]
sonarcloud[bot] commented on PR #5196: URL: https://github.com/apache/hive/pull/5196#issuecomment-2151409311 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5196) **Quality Gate passed** Issues  [14 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5196=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5196=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5196=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5196) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28306: Iceberg: Return new scan after applying column project parameter [hive]
sonarcloud[bot] commented on PR #5282: URL: https://github.com/apache/hive/pull/5282#issuecomment-2151371282 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5282) **Quality Gate passed** Issues  [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive=5282=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5282=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5282=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5282) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28306: Iceberg: Return new scan after applying column project parameter [hive]
zhangbutao commented on code in PR #5282:
URL: https://github.com/apache/hive/pull/5282#discussion_r1628711004
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInternalRecordWrapper.java:
##
@@ -57,7 +57,7 @@ public IcebergInternalRecordWrapper(StructType tableSchema,
StructType readSchem
public IcebergInternalRecordWrapper wrap(StructLike record) {
int idx = 0;
for (Types.NestedField field : readSchema.fields()) {
- int position = fieldToPositionInTableSchema.get(field.name());
+ int position = fieldToPositionInReadSchema.get(field.name());
Review Comment:
Fixed the failed metadata qtest:
http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5282/1/tests/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
dengzhhu653 merged PR #5278: URL: https://github.com/apache/hive/pull/5278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28294: drop database cascade operation can skip client side filt… [hive]
dengzhhu653 commented on code in PR #5275:
URL: https://github.com/apache/hive/pull/5275#discussion_r1628630566
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/drop/DropDatabaseAnalyzer.java:
##
@@ -73,6 +74,11 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
}
outputs.add(new WriteEntity(table, lockType));
}
+// fetch all the functions in the database
+List functions = db.getFunctionsInDb(databaseName, ".*");
Review Comment:
On HS2 restart, we load all functions from HMS,
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L364
I think this call is much light weight than the `getAllTableObjects` in this
method.
Another point is the `drop database` is a long consuming call as expected
and less frequent, I'm a little worry about if it's worth to add an extra API
just to fetch all the functions in this database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28294: drop database cascade operation can skip client side filt… [hive]
dengzhhu653 commented on code in PR #5275:
URL: https://github.com/apache/hive/pull/5275#discussion_r1628622529
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/drop/DropDatabaseAnalyzer.java:
##
@@ -60,7 +62,13 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
// if cascade=true, then we need to authorize the drop table action as
well, and add the tables to the outputs
boolean isDbLevelLock = true;
if (cascade) {
+ SessionState sessionState = SessionState.get();
try {
+HiveConf hiveConf = new HiveConf(conf);
+hiveConf.set("hive.metastore.client.filter.enabled", "false");
+if (sessionState != null) {
Review Comment:
Not sure if the current `db` would take this property, as this `db` is
created by the original `conf` which doesn't have this property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-26547: setting primary key constraint for DELEGATION_TOKENS tabl… [hive]
github-actions[bot] commented on PR #3942: URL: https://github.com/apache/hive/pull/3942#issuecomment-2151160108 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27854:Driver compile may causing memory leak [hive]
github-actions[bot] commented on PR #4854: URL: https://github.com/apache/hive/pull/4854#issuecomment-2151160087 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] Fixed 8 flaky tests in `hbase-handler` [hive]
github-actions[bot] commented on PR #4900: URL: https://github.com/apache/hive/pull/4900#issuecomment-2151160035 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28294: drop database cascade operation can skip client side filt… [hive]
sonarcloud[bot] commented on PR #5275: URL: https://github.com/apache/hive/pull/5275#issuecomment-2151090462 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5275) **Quality Gate passed** Issues  [18 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5275=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5275=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5275=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5275) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28305: ProfileServlet: add html to output formats [hive]
sonarcloud[bot] commented on PR #5281: URL: https://github.com/apache/hive/pull/5281#issuecomment-2151000564 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5281) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5281=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5281=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5281=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5281) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28294: drop database cascade operation can skip client side filt… [hive]
saihemanth-cloudera commented on code in PR #5275:
URL: https://github.com/apache/hive/pull/5275#discussion_r1628258033
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/drop/DropDatabaseAnalyzer.java:
##
@@ -73,6 +74,11 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
}
outputs.add(new WriteEntity(table, lockType));
}
+// fetch all the functions in the database
+List functions = db.getFunctionsInDb(databaseName, ".*");
Review Comment:
We cannot take it for granted for all the customer use cases. If one of the
database end up having lot of functions then it becomes a bottle neck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] WIP: Dont review [hive]
sonarcloud[bot] commented on PR #5012: URL: https://github.com/apache/hive/pull/5012#issuecomment-2150580731 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5012) **Quality Gate passed** Issues  [450 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5012=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5012=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5012=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5012) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28306: Iceberg: Return new scan after applying column project parameter [hive]
sonarcloud[bot] commented on PR #5282: URL: https://github.com/apache/hive/pull/5282#issuecomment-2150537711 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5282) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5282=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5282=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5282=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5282) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
[PR] HIVE-28306: Iceberg: Return new scan after applying column project parameter [hive]
zhangbutao opened a new pull request, #5282: URL: https://github.com/apache/hive/pull/5282 ### What changes were proposed in this pull request? 1. shoud return a new scan applying column project parameter, Otherwise it doesn't make any sense 2. `scan.project()` and `scan.select()` can not be specified at the same time Check this: https://github.com/apache/iceberg/pull/1353/files#diff-a94dee2a22a2890689a84529a978a769bdc3b084f35d79caf9608d49b2e1cc1b ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? No ### Is the change a dependency upgrade? No ### How was this patch tested? Existing tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
[PR] HIVE-28305: ProfileServlet: add html to output formats [hive]
abstractdog opened a new pull request, #5281: URL: https://github.com/apache/hive/pull/5281 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### Is the change a dependency upgrade? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28299: Iceberg: Optimize show partitions through column projection [hive]
sonarcloud[bot] commented on PR #5276: URL: https://github.com/apache/hive/pull/5276#issuecomment-2150339359 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5276) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5276=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5276=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5276=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5276) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27498: Support custom delimiter in SkippingTextInputFormat [hive]
sonarcloud[bot] commented on PR #5265: URL: https://github.com/apache/hive/pull/5265#issuecomment-2149832350 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5265) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5265=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5265) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28082: HiveAggregateReduceFunctionsRule could generate an inconsistent result [hive]
kasakrisz merged PR #5091: URL: https://github.com/apache/hive/pull/5091 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28304: Replace org.apache.log4j.Logger with org.slf4j.Logger [hive]
sonarcloud[bot] commented on PR #5280: URL: https://github.com/apache/hive/pull/5280#issuecomment-2149688240 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5280) **Quality Gate passed** Issues  [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5280=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5280=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5280=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5280) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
sonarcloud[bot] commented on PR #5278: URL: https://github.com/apache/hive/pull/5278#issuecomment-2149683377 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5278) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5278=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5278=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5278=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5278) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28303: Develocity integration [hive]
sonarcloud[bot] commented on PR #5279: URL: https://github.com/apache/hive/pull/5279#issuecomment-2149506611 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5279) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5279=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5279=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5279=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5279) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27498: Support custom delimiter in SkippingTextInputFormat [hive]
sonarcloud[bot] commented on PR #5265: URL: https://github.com/apache/hive/pull/5265#issuecomment-2149364284 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5265) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5265=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5265) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
[PR] HIVE-28304: Replace org.apache.log4j.Logger with org.slf4j.Logger [hive]
ngsg opened a new pull request, #5280: URL: https://github.com/apache/hive/pull/5280 ### What changes were proposed in this pull request? Replace org.apache.log4j.Logger with org.slf4j.Logger. ### Why are the changes needed? Logger should be created via facade(SLF4J), but some classes still directly access to Logger implementation. This patch fixes this issue. ### Does this PR introduce _any_ user-facing change? No ### Is the change a dependency upgrade? No ### How was this patch tested? This patch was not tested as it just changes logger instantiation code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r1627388192
##
ql/src/test/queries/clientpositive/udf_configurable.q:
##
@@ -0,0 +1,55 @@
+CREATE TABLE test_udf_configurable (cint1 INT, cint2 INT, ctimestamp
TIMESTAMP, text_timestamp STRING);
+INSERT INTO test_udf_configurable VALUES
+ (1, 3, CAST('1970-01-01 01:02:03' AS TIMESTAMP), '1970-01-01 01:02:03
4'),
+ (2, 5, CAST('1970-01-02 04:05:06' AS TIMESTAMP), '1970-01-02 04:05:06
5'),
+ (3, 7, CAST('1970-01-03 07:08:09' AS TIMESTAMP), '1970-01-03 07:08:09
6');
+
+set hive.compat=latest;
+set hive.strict.timestamp.conversion=false;
+set hive.int.timestamp.conversion.in.seconds=true;
+set hive.local.time.zone=Asia/Bangkok;
+set hive.datetime.formatter=SIMPLE;
+set hive.masking.algo=sha512;
+set hive.use.googleregex.engine=true;
+
+-- On HiveServer2
+SELECT
+ -- DECIMAL because of hive.compat=latest
+ TYPEOF(cint1 / cint2),
Review Comment:
I pushed the change and am waiting for CI.
https://github.com/apache/hive/pull/5278/commits/db8edc83ae2e0ba44c607196c61ed7e60c513561
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r1627387403
##
ql/src/test/queries/clientpositive/udf_configurable.q:
##
@@ -0,0 +1,55 @@
+CREATE TABLE test_udf_configurable (cint1 INT, cint2 INT, ctimestamp
TIMESTAMP, text_timestamp STRING);
+INSERT INTO test_udf_configurable VALUES
+ (1, 3, CAST('1970-01-01 01:02:03' AS TIMESTAMP), '1970-01-01 01:02:03
4'),
+ (2, 5, CAST('1970-01-02 04:05:06' AS TIMESTAMP), '1970-01-02 04:05:06
5'),
+ (3, 7, CAST('1970-01-03 07:08:09' AS TIMESTAMP), '1970-01-03 07:08:09
6');
+
+set hive.compat=latest;
+set hive.strict.timestamp.conversion=false;
+set hive.int.timestamp.conversion.in.seconds=true;
+set hive.local.time.zone=Asia/Bangkok;
+set hive.datetime.formatter=SIMPLE;
+set hive.masking.algo=sha512;
+set hive.use.googleregex.engine=true;
+
+-- On HiveServer2
+SELECT
+ -- DECIMAL because of hive.compat=latest
+ TYPEOF(cint1 / cint2),
Review Comment:
As you say, I found we can simply remove TYPEOF since the text expression or
precision of DOUBLE and DECIMAL is different. This is the diff between
`hive.compat=latest` vs default.
```
< .5 false 1970-01-01 02:46:40 1970-01-01 01:02:03 4
1970-01-01 09:46:40 4 -21477 -21477
ca764b57c635a893c91b0edaea84bca142e065990e1a66b9c60cf855777af51070444210d30e359fa81e5a77c68a073224e0c7343d957f556816618a10baa37c
false
< 4000.0 false 1970-01-01 05:33:20 1970-01-02 04:05:06 5
1970-01-01 12:33:20 4 75906 75906
7882085a4ed36e7c675fcd12083eefa9a208f1fdcfba10b3a986bb5b1d43e23da3ba87b3933feebfed80ba9e05e4e73f72231cb8022453e562e034e8b5c603b9
false
< 4285.714285714285 false 1970-01-01 08:20:00 1970-01-03 07:08:09 6
1970-01-01 15:20:00 4 173289 173289
d9899e12fce35a4f65cd63aee1e953dacd9e91229d52dd17ddad8e44c17d2d081c0a5519a96637663bc6baa2d7c96d6f1a9d6944ac7668c7926ebc10c9f3b81b
false
---
> .333 false 1970-01-01 02:46:40 1970-01-01 01:02:03 4
1970-01-01 09:46:40 4 -21477 -21477
ca764b57c635a893c91b0edaea84bca142e065990e1a66b9c60cf855777af51070444210d30e359fa81e5a77c68a073224e0c7343d957f556816618a10baa37c
false
> 4000.000 false 1970-01-01 05:33:20 1970-01-02 04:05:06 5
1970-01-01 12:33:20 4 75906 75906
7882085a4ed36e7c675fcd12083eefa9a208f1fdcfba10b3a986bb5b1d43e23da3ba87b3933feebfed80ba9e05e4e73f72231cb8022453e562e034e8b5c603b9
false
> 4285.71428571429 false 1970-01-01 08:20:00 1970-01-03 07:08:09 6
1970-01-01 15:20:00 4 173289 173289
d9899e12fce35a4f65cd63aee1e953dacd9e91229d52dd17ddad8e44c17d2d081c0a5519a96637663bc6baa2d7c96d6f1a9d6944ac7668c7926ebc10c9f3b81b
false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
[PR] HIVE-28303: Develocity integration [hive]
ribafish opened a new pull request, #5279: URL: https://github.com/apache/hive/pull/5279 @simhadri-g, it was nice meeting you at Community over Code yesterday. This PR will enable you to publish Build Scans to [ge.apache.org](https://ge.apache.org/) as discussed. https://issues.apache.org/jira/browse/HIVE-28303 ### What changes were proposed in this pull request? This PR publishes a build scan for every CI build and for every local build from an authenticated Apache committer. The build will not fail if publishing fails. Local caching on CI and remote caching was left disabled on this PR by design so that the build is not affected by this change. The build scans of the Apache Hive project are published to the Develocity instance at [ge.apache.org](https://ge.apache.org/), hosted by the Apache Software Foundation and run in partnership between the ASF and Gradle. This Develocity instance has all features and extensions enabled and is freely available for use by the Apache Hive project and all other Apache projects. On this Develocity instance, Apache Hive will have access not only to all of the published build scans but other aggregate data features such as: - Dashboards to view all historical build scans, along with performance trends over time - Build failure analytics for enhanced investigation and diagnosis of build failures - Test failure analytics to better understand trends and causes around slow, failing, and flaky tests A licensing agreement between ASF and Gradle is already in place, covering the contribution. For more information, you can reach out to the ASF Infra team. Please let me know if there are any questions about the value of Develocity or the changes in this pull request and I’d be happy to address them. ### Does this PR introduce _any_ user-facing change? No ### Is the change a dependency upgrade? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27972: Set 'tez' as default value in hive.execution.engine [hive]
abstractdog commented on code in PR #4973:
URL: https://github.com/apache/hive/pull/4973#discussion_r1627290418
##
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##
@@ -4564,7 +4564,7 @@ public static enum ConfVars {
HIVE_DECODE_PARTITION_NAME("hive.decode.partition.name", false,
"Whether to show the unquoted partition names in query results."),
-HIVE_EXECUTION_ENGINE("hive.execution.engine", "mr", new StringSet(true,
"mr", "tez"),
+HIVE_EXECUTION_ENGINE("hive.execution.engine", "tez", new StringSet(true,
"tez", "mr"),
"Chooses execution engine. Options are: mr (Map reduce, default), tez.
While MR\n" +
Review Comment:
this has been addressed in recent commits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28299: Iceberg: Optimize show partitions through column projection [hive]
zhangbutao commented on code in PR #5276: URL: https://github.com/apache/hive/pull/5276#discussion_r1627272182 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -1840,7 +1841,8 @@ private List getPartitions(DDLOperationContext context, Configuration jo fetcher.initialize(job, HiveTableUtil.getSerializationProps()); org.apache.hadoop.hive.ql.metadata.Table metaDataPartTable = context.getDb().getTable(hmstbl.getDbName(), hmstbl.getTableName(), "partitions", true); - Deserializer currSerDe = metaDataPartTable.getDeserializer(); + Deserializer currSerDe = HiveMetaStoreUtils.getDeserializer(job, metaDataPartTable.getTTable(), Review Comment: Use job conf to get the iceberg Deserializer so that we can propagate `ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR` to `HiveIcebergSerDe` https://github.com/apache/hive/blob/e570e65a397aae47c62472f12acaaed06b68cf54/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java#L163 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28294: drop database cascade operation can skip client side filt… [hive]
dengzhhu653 commented on PR #5275: URL: https://github.com/apache/hive/pull/5275#issuecomment-2149210981 I'm trying to understand the problem, why we need "all the tables can be authorized for drop database cascade", given that we have the drop privilege on the database? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28299: Iceberg: Optimize show partitions through column projection [hive]
zhangbutao commented on code in PR #5276: URL: https://github.com/apache/hive/pull/5276#discussion_r1625504062 ## iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java: ## @@ -1840,7 +1841,9 @@ private List getPartitions(DDLOperationContext context, Configuration jo fetcher.initialize(job, HiveTableUtil.getSerializationProps()); org.apache.hadoop.hive.ql.metadata.Table metaDataPartTable = context.getDb().getTable(hmstbl.getDbName(), hmstbl.getTableName(), "partitions", true); + SessionState.getSessionConf().set(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR, "partition,spec_id"); Review Comment: https://github.com/apache/hive/blob/45867be6cb5308566e4cf16c7b4cf8081085b58c/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java#L343-L344 `HiveIcebergSerDe `can only use the `session conf`, so here i set the `ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR` in the session conf, and then unset this propertity to avoid affecting other sql queries in the current seesion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28294: drop database cascade operation can skip client side filt… [hive]
dengzhhu653 commented on code in PR #5275:
URL: https://github.com/apache/hive/pull/5275#discussion_r1627250255
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/drop/DropDatabaseAnalyzer.java:
##
@@ -73,6 +74,11 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
}
outputs.add(new WriteEntity(table, lockType));
}
+// fetch all the functions in the database
+List functions = db.getFunctionsInDb(databaseName, ".*");
Review Comment:
Given in reality, the number of permanent functions stored in remote db
should be smaller compared to the other meta, I think we can use
`db.getAllFunctions` then select the functions belonging to this database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28294: drop database cascade operation can skip client side filt… [hive]
dengzhhu653 commented on code in PR #5275:
URL: https://github.com/apache/hive/pull/5275#discussion_r1627242981
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/database/drop/DropDatabaseAnalyzer.java:
##
@@ -61,7 +62,7 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
boolean isDbLevelLock = true;
if (cascade) {
try {
-List tables = db.getAllTableObjects(databaseName);
+List tables = db.getAllTableObjects(databaseName, true);
Review Comment:
There is a `hive.metastore.client.filter.enabled` to disable the client
filter, we can use it instead of depend on a new method, e.g
```java
try {
HiveConf hiveConf = new HiveConf(conf);
conf.set("hive.metastore.client.filter.enabled", "false");
Hive db = Hive.get(hiveConf);
} finally {
// restore
db = Hive.get(conf);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28286: Add filtering support for get_table_metas (Naveen Gangam) [hive]
sonarcloud[bot] commented on PR #5269: URL: https://github.com/apache/hive/pull/5269#issuecomment-2149123329 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5269) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5269=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5269=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5269=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5269) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
dengzhhu653 closed pull request #4851: HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited URL: https://github.com/apache/hive/pull/4851 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27498: Support custom delimiter in SkippingTextInputFormat [hive]
sonarcloud[bot] commented on PR #5265: URL: https://github.com/apache/hive/pull/5265#issuecomment-2149075305 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5265) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5265=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5265) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r1627119747
##
ql/src/test/queries/clientpositive/udf_configurable.q:
##
@@ -0,0 +1,55 @@
+CREATE TABLE test_udf_configurable (cint1 INT, cint2 INT, ctimestamp
TIMESTAMP, text_timestamp STRING);
+INSERT INTO test_udf_configurable VALUES
+ (1, 3, CAST('1970-01-01 01:02:03' AS TIMESTAMP), '1970-01-01 01:02:03
4'),
+ (2, 5, CAST('1970-01-02 04:05:06' AS TIMESTAMP), '1970-01-02 04:05:06
5'),
+ (3, 7, CAST('1970-01-03 07:08:09' AS TIMESTAMP), '1970-01-03 07:08:09
6');
+
+set hive.compat=latest;
+set hive.strict.timestamp.conversion=false;
+set hive.int.timestamp.conversion.in.seconds=true;
+set hive.local.time.zone=Asia/Bangkok;
+set hive.datetime.formatter=SIMPLE;
+set hive.masking.algo=sha512;
+set hive.use.googleregex.engine=true;
+
+-- On HiveServer2
+SELECT
+ -- DECIMAL because of hive.compat=latest
+ TYPEOF(cint1 / cint2),
Review Comment:
You are right! I wonder what is the most reasonable way to identify if it is
DOUBLE or DECIMAL...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28194: Regenerate HS2 thrift port in TestRetryingThriftCLIServiceClient [hive]
armitage420 commented on PR #5205: URL: https://github.com/apache/hive/pull/5205#issuecomment-2149048114 @deniskuzZ I have made the required changes, I hope you could review the PR when possible! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r1627115225
##
ql/src/test/queries/clientpositive/udf_configurable.q:
##
@@ -0,0 +1,55 @@
+CREATE TABLE test_udf_configurable (cint1 INT, cint2 INT, ctimestamp
TIMESTAMP, text_timestamp STRING);
+INSERT INTO test_udf_configurable VALUES
+ (1, 3, CAST('1970-01-01 01:02:03' AS TIMESTAMP), '1970-01-01 01:02:03
4'),
+ (2, 5, CAST('1970-01-02 04:05:06' AS TIMESTAMP), '1970-01-02 04:05:06
5'),
+ (3, 7, CAST('1970-01-03 07:08:09' AS TIMESTAMP), '1970-01-03 07:08:09
6');
+
+set hive.compat=latest;
+set hive.strict.timestamp.conversion=false;
+set hive.int.timestamp.conversion.in.seconds=true;
+set hive.local.time.zone=Asia/Bangkok;
+set hive.datetime.formatter=SIMPLE;
+set hive.masking.algo=sha512;
+set hive.use.googleregex.engine=true;
+
+-- On HiveServer2
+SELECT
+ -- DECIMAL because of hive.compat=latest
+ TYPEOF(cint1 / cint2),
+ -- Allowed by hive.strict.timestamp.conversion=false
+ cint1 = ctimestamp,
Review Comment:
@dengzhhu653 This is checking this validation if I remember correctly.
https://github.com/apache/hive/blob/e570e65a397aae47c62472f12acaaed06b68cf54/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseCompare.java#L178-L190
```
0: jdbc:hive2://hive-hiveserver2:1/defaul> SELECT 100 = CAST('2024-01-01
00:00:00' AS TIMESTAMP);
Error: Error while compiling statement: FAILED: SemanticException [Error
10014]: Line 1:7 Wrong arguments ''2024-01-01 00:00:00'': Casting
DATE/TIMESTAMP to NUMERIC is prohibited (hive.strict.timestamp.conversion)
(state=42000,code=10014)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
dengzhhu653 commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r162703
##
ql/src/test/queries/clientpositive/udf_configurable.q:
##
@@ -0,0 +1,55 @@
+CREATE TABLE test_udf_configurable (cint1 INT, cint2 INT, ctimestamp
TIMESTAMP, text_timestamp STRING);
+INSERT INTO test_udf_configurable VALUES
+ (1, 3, CAST('1970-01-01 01:02:03' AS TIMESTAMP), '1970-01-01 01:02:03
4'),
+ (2, 5, CAST('1970-01-02 04:05:06' AS TIMESTAMP), '1970-01-02 04:05:06
5'),
+ (3, 7, CAST('1970-01-03 07:08:09' AS TIMESTAMP), '1970-01-03 07:08:09
6');
+
+set hive.compat=latest;
+set hive.strict.timestamp.conversion=false;
+set hive.int.timestamp.conversion.in.seconds=true;
+set hive.local.time.zone=Asia/Bangkok;
+set hive.datetime.formatter=SIMPLE;
+set hive.masking.algo=sha512;
+set hive.use.googleregex.engine=true;
+
+-- On HiveServer2
+SELECT
+ -- DECIMAL because of hive.compat=latest
+ TYPEOF(cint1 / cint2),
Review Comment:
nit: this `TYPEOF(cint1 / cint2)` happens only at compile stage, should we
change it to `cint1 / cint2`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
dengzhhu653 commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r1627098172
##
ql/src/test/queries/clientpositive/udf_configurable.q:
##
@@ -0,0 +1,55 @@
+CREATE TABLE test_udf_configurable (cint1 INT, cint2 INT, ctimestamp
TIMESTAMP, text_timestamp STRING);
+INSERT INTO test_udf_configurable VALUES
+ (1, 3, CAST('1970-01-01 01:02:03' AS TIMESTAMP), '1970-01-01 01:02:03
4'),
+ (2, 5, CAST('1970-01-02 04:05:06' AS TIMESTAMP), '1970-01-02 04:05:06
5'),
+ (3, 7, CAST('1970-01-03 07:08:09' AS TIMESTAMP), '1970-01-03 07:08:09
6');
+
+set hive.compat=latest;
+set hive.strict.timestamp.conversion=false;
+set hive.int.timestamp.conversion.in.seconds=true;
+set hive.local.time.zone=Asia/Bangkok;
+set hive.datetime.formatter=SIMPLE;
+set hive.masking.algo=sha512;
+set hive.use.googleregex.engine=true;
+
+-- On HiveServer2
+SELECT
+ -- DECIMAL because of hive.compat=latest
+ TYPEOF(cint1 / cint2),
+ -- Allowed by hive.strict.timestamp.conversion=false
+ cint1 = ctimestamp,
Review Comment:
nit: the left side of `cint1 = ctimestamp` shouldn't be converted to
timestamp, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27855: Create external tables for Hive and Tez protologging events [hive]
sonarcloud[bot] commented on PR #5036: URL: https://github.com/apache/hive/pull/5036#issuecomment-2148978035 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5036) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5036=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5036=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5036=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5036) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28082: HiveAggregateReduceFunctionsRule could generate an inconsistent result [hive]
okumin commented on PR #5091: URL: https://github.com/apache/hive/pull/5091#issuecomment-2148967922 CI passed. > as you mentioned: "I believe SUM should return NULL if all rows are evaluated as NULL" on both CBO and non-CBO path. Swapping these lines may help to solve this: BTW, I filed another ticket for this change. https://issues.apache.org/jira/browse/HIVE-28302 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28082: HiveAggregateReduceFunctionsRule could generate an inconsistent result [hive]
okumin commented on code in PR #5091: URL: https://github.com/apache/hive/pull/5091#discussion_r1627007333 ## ql/src/test/results/clientpositive/llap/udaf_number_format.q.out: ## @@ -92,4 +92,4 @@ FROM src POSTHOOK: type: QUERY POSTHOOK: Input: default@src A masked pattern was here -0.00.0 NULLNULL +0.0NULLNULLNULL Review Comment: I believe this diff is expected based on the discussion. https://github.com/apache/hive/blob/e78e27bc593c20a312c76b7ba9c06b96176bc647/ql/src/test/queries/clientpositive/udaf_number_format.q -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on PR #5278: URL: https://github.com/apache/hive/pull/5278#issuecomment-2148911365 This seems to be unrelated and temporal. https://github.com/apache/hive/actions/runs/9376224477/job/25815816131?pr=5278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
sonarcloud[bot] commented on PR #5278: URL: https://github.com/apache/hive/pull/5278#issuecomment-2148693719 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5278) **Quality Gate passed** Issues  [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5278=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5278=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5278=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5278) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on PR #4851: URL: https://github.com/apache/hive/pull/4851#issuecomment-2148635867 @dengzhhu653 I created a new PR because I have no permission to directly update @basapuram-kumar 's branch. https://github.com/apache/hive/pull/5278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r1626781394
##
ql/src/test/queries/clientpositive/udf_configurable.q:
##
@@ -0,0 +1,55 @@
+CREATE TABLE test_udf_configurable (cint1 INT, cint2 INT, ctimestamp
TIMESTAMP, text_timestamp STRING);
+INSERT INTO test_udf_configurable VALUES
+ (1, 3, CAST('1970-01-01 01:02:03' AS TIMESTAMP), '1970-01-01 01:02:03
4'),
+ (2, 5, CAST('1970-01-02 04:05:06' AS TIMESTAMP), '1970-01-02 04:05:06
5'),
+ (3, 7, CAST('1970-01-03 07:08:09' AS TIMESTAMP), '1970-01-03 07:08:09
6');
+
+set hive.compat=latest;
+set hive.strict.timestamp.conversion=false;
+set hive.int.timestamp.conversion.in.seconds=true;
+set hive.local.time.zone=Asia/Bangkok;
+set hive.datetime.formatter=SIMPLE;
+set hive.masking.algo=sha512;
+set hive.use.googleregex.engine=true;
+
+-- On HiveServer2
+SELECT
+ -- DECIMAL because of hive.compat=latest
+ TYPEOF(cint1 / cint2),
+ -- Allowed by hive.strict.timestamp.conversion=false
+ cint1 = ctimestamp,
+ -- Allowed by hive.strict.timestamp.conversion=false
+ -- Interpreted as seconds because of
hive.int.timestamp.conversion.in.seconds=true
+ CAST(cint1 AS TIMESTAMP),
+ -- The semantics of "u" is different between SimpleDateFormat and
DateTimeFormatter
+ DATE_FORMAT(ctimestamp, '-MM-dd HH:mm:ss u'),
+ FROM_UNIXTIME(cint1, '-MM-dd HH:mm:ss u'),
+ TO_UNIX_TIMESTAMP(text_timestamp, '-MM-dd HH:mm:ss u'),
+ UNIX_TIMESTAMP(text_timestamp, '-MM-dd HH:mm:ss u'),
+ -- SHA512 is used
+ MASK_HASH(text_timestamp),
+ -- Java's Pattern doesn't support it, then it fails with
hive.use.googleregex.engine=false
+ text_timestamp RLIKE '\\p{Katakana}+'
Review Comment:
The following files would touch SessionState or HiveConf.
```
git grep -l -e 'org.apache.hadoop.hive.ql.session.SessionState' -e
'org.apache.hadoop.hive.conf.HiveConf' | grep
'ql/src/java/org/apache/hadoop/hive/ql/udf/generic'
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseArithmetic.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseCompare.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseNumeric.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentAuthorizer.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentCatalog.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentDatabase.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentDate.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentGroups.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentSchema.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentTimestamp.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCurrentUser.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateFormat.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUnixTime.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLoggedInUser.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMaskHash.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPDivide.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRegExp.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRestrictInformationSchema.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToUnixTimeStamp.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUnixTimeStamp.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSQLSchema.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/InstantFormatter.java
```
I didn't add tests for the following UDFs because they are marked as
"runtime constant" and the issue will unlikely happen. As another point, it is
hard to configure some of them in qtest.
```
CURRENT_AUTHORIZER(),
CURRENT_CATALOG(),
CURRENT_DATABASE(),
CURRENT_DATE(),
CURRENT_GROUPS(),
CURRENT_SCHEMA(),
CURRENT_TIMESTAMP(),
CURRENT_USER(),
LOGGED_IN_USER(),
RESTRICT_INFORMATION_SCHEMA()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r1626780141
##
ql/src/test/queries/clientpositive/non_strict_numeric_to_timestamp_conversion.q:
##
@@ -4,6 +4,8 @@ create table test_num_ts_input(begin string, ts string);
insert into test_num_ts_input
values('1653209895687','2022-05-22T15:58:15.931+07:00'),('1653209938316','2022-05-22T15:58:58.490+07:00'),('1653209962021','2022-05-22T15:59:22.191+07:00'),('1653210021993','2022-05-22T16:00:22.174+07:00');
+set hive.vectorized.execution.enabled=false;
Review Comment:
The problem was not reproduced without this session param.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r1626779353
##
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java:
##
@@ -71,32 +71,38 @@ public class GenericUDFTimestamp extends GenericUDF {
* otherwise, it's interpreted as timestamp in seconds.
*/
private boolean intToTimestampInSeconds = false;
- private boolean strict = true;
@Override
public ObjectInspector initialize(ObjectInspector[] arguments) throws
UDFArgumentException {
checkArgsSize(arguments, 1, 1);
checkArgPrimitive(arguments, 0);
checkArgGroups(arguments, 0, tsInputTypes, STRING_GROUP, DATE_GROUP,
NUMERIC_GROUP, VOID_GROUP, BOOLEAN_GROUP);
-strict = SessionState.get() != null ? SessionState.get().getConf()
-.getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION) : new HiveConf()
-.getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION);
-intToTimestampInSeconds = SessionState.get() != null ?
SessionState.get().getConf()
-.getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS) : new
HiveConf()
-.getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS);
-
-if (strict) {
- if (PrimitiveObjectInspectorUtils.getPrimitiveGrouping(tsInputTypes[0])
== PrimitiveGrouping.NUMERIC_GROUP) {
+final SessionState ss = SessionState.get();
+if (ss != null) {
+ final boolean strict =
ss.getConf().getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION);
+ final PrimitiveGrouping grouping =
PrimitiveObjectInspectorUtils.getPrimitiveGrouping(tsInputTypes[0]);
+ if (strict && grouping == PrimitiveGrouping.NUMERIC_GROUP) {
Review Comment:
I expect this validation is required only at compile-time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
okumin commented on code in PR #5278:
URL: https://github.com/apache/hive/pull/5278#discussion_r1626777303
##
hbase-handler/src/test/queries/positive/hbase_timestamp.q:
##
@@ -1,4 +1,6 @@
--! qt:dataset:src
+set hive.strict.timestamp.conversion=false;
Review Comment:
Because of the removal of the param from `data/conf/hive-site.xml`.
##
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorFilterCompare.java:
##
@@ -84,6 +84,7 @@ public TestVectorFilterCompare() {
// Arithmetic operations rely on getting conf from SessionState, need to
initialize here.
SessionState ss = new SessionState(new HiveConf());
ss.getConf().setVar(HiveConf.ConfVars.HIVE_COMPAT, "latest");
+
ss.getConf().setBoolVar(HiveConf.ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION,
false);
Review Comment:
Because of the removal of the param from `data/conf/hive-site.xml`.
##
data/conf/hive-site.xml:
##
@@ -326,11 +326,6 @@
false
-
- hive.strict.timestamp.conversion
- false
-
Review Comment:
Test HS2 clusters inherit this hive-site.xml, then this param is stored in
where `new HiveConf()` picks up.
In order to make sure that test cases use the param not in hive-site but in
session params, I'd like to remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28294: drop database cascade operation can skip client side filt… [hive]
sonarcloud[bot] commented on PR #5275: URL: https://github.com/apache/hive/pull/5275#issuecomment-2148552221 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5275) **Quality Gate passed** Issues  [22 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5275=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5275=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5275=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5275) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27498: Support custom delimiter in SkippingTextInputFormat [hive]
sonarcloud[bot] commented on PR #5265: URL: https://github.com/apache/hive/pull/5265#issuecomment-2148412960 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5265) **Quality Gate passed** Issues  [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5265=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5265) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28194: Regenerate HS2 thrift port in TestRetryingThriftCLIServiceClient [hive]
sonarcloud[bot] commented on PR #5205: URL: https://github.com/apache/hive/pull/5205#issuecomment-2148255272 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5205) **Quality Gate passed** Issues  [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5205=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5205=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5205=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5205) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
sonarcloud[bot] commented on PR #5273: URL: https://github.com/apache/hive/pull/5273#issuecomment-2148093437 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5273) **Quality Gate passed** Issues  [39 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5273=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5273=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5273=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5273) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] [WIP] HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited [hive]
sonarcloud[bot] commented on PR #5278: URL: https://github.com/apache/hive/pull/5278#issuecomment-2148036035 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5278) **Quality Gate passed** Issues  [26 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5278=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5278=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5278=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5278) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626333863
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java:
##
@@ -78,20 +75,22 @@ private void showTables() throws HiveException {
}
private void showTablesExtended() throws HiveException {
-List tableObjects = new ArrayList<>();
-tableObjects.addAll(context.getDb().getTableObjects(desc.getDbName(),
null, desc.getTypeFilter()));
-if (desc.getPattern() != null) {
- Pattern pattern =
Pattern.compile(UDFLike.likePatternToRegExp(desc.getPattern()),
Pattern.CASE_INSENSITIVE);
- tableObjects = tableObjects.stream()
- .filter(object -> pattern.matcher(object.getTableName()).matches())
- .collect(Collectors.toList());
+List> tableTypePairs = new ArrayList<>();
Review Comment:
Good idea, addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626328981
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##
@@ -2897,26 +2897,44 @@ public List getTables(String dbname, String
tablePattern) throws MetaExc
@Override
public List getTables(String catName, String dbName, String
tablePattern)
throws TException {
-List tables = new ArrayList<>();
-GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
-projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner",
"ownerType"));
-GetTablesRequest req = new GetTablesRequest(dbName);
-req.setCatName(catName);
-req.setCapabilities(version);
-req.setTblNames(null);
-if(tablePattern == null){
+if (tablePattern == null) {
tablePattern = ".*";
}
-req.setTablesPattern(tablePattern);
-if (processorCapabilities != null)
- req.setProcessorCapabilities(new
ArrayList(Arrays.asList(processorCapabilities)));
-if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
-req.setProjectionSpec(projectionsSpec);
-List tableObjects =
client.get_table_objects_by_name_req(req).getTables();
-tableObjects =
deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled,
filterHook, tableObjects));
-for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+List tables = new ArrayList<>();
+Database db = null;
+try {
+ db = getDatabase(catName, dbName);
+} catch (Exception e) { /* appears exception is not thrown currently if db
doesnt exist */ }
+
+if (MetaStoreUtils.isDatabaseRemote(db)) {
+ // TODO: remote database does not support list table names by pattern
yet.
+ // This branch can be removed once it's supported.
+ GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
+ projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName",
"owner", "ownerType"));
+ GetTablesRequest req = new GetTablesRequest(dbName);
+ req.setCatName(catName);
+ req.setCapabilities(version);
+ req.setTblNames(null);
+ req.setTablesPattern(tablePattern);
+ if (processorCapabilities != null)
+req.setProcessorCapabilities(new
ArrayList(Arrays.asList(processorCapabilities)));
Review Comment:
It's copied from source, removed redundant ArrayList.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626324311
##
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.1.0.derby.sql:
##
@@ -269,7 +269,7 @@ CREATE UNIQUE INDEX "APP"."ROLEENTITYINDEX" ON
"APP"."ROLES" ("ROLE_NAME");
CREATE INDEX "APP"."TABLEPRIVILEGEINDEX" ON "APP"."TBL_PRIVS" ("AUTHORIZER",
"TBL_ID", "PRINCIPAL_NAME", "PRINCIPAL_TYPE", "TBL_PRIV", "GRANTOR",
"GRANTOR_TYPE");
-CREATE UNIQUE INDEX "APP"."UNIQUETABLE" ON "APP"."TBLS" ("TBL_NAME", "DB_ID");
+CREATE UNIQUE INDEX "APP"."UNIQUETABLE" ON "APP"."TBLS" ("DB_ID", "TBL_NAME");
Review Comment:
We change the order of this composite index to improve performance. Because
we always get the`DB_ID` first while doing any query for hive table, so the
first level (DB_ID) of the index alway take effect after this change.
It's a similar change like
https://github.com/apache/hive/pull/4831#pullrequestreview-1814098075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626315585
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##
@@ -2897,26 +2897,44 @@ public List getTables(String dbname, String
tablePattern) throws MetaExc
@Override
public List getTables(String catName, String dbName, String
tablePattern)
throws TException {
-List tables = new ArrayList<>();
-GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
-projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner",
"ownerType"));
-GetTablesRequest req = new GetTablesRequest(dbName);
-req.setCatName(catName);
-req.setCapabilities(version);
-req.setTblNames(null);
-if(tablePattern == null){
+if (tablePattern == null) {
tablePattern = ".*";
}
-req.setTablesPattern(tablePattern);
-if (processorCapabilities != null)
- req.setProcessorCapabilities(new
ArrayList(Arrays.asList(processorCapabilities)));
-if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
-req.setProjectionSpec(projectionsSpec);
-List tableObjects =
client.get_table_objects_by_name_req(req).getTables();
-tableObjects =
deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled,
filterHook, tableObjects));
-for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+List tables = new ArrayList<>();
+Database db = null;
+try {
+ db = getDatabase(catName, dbName);
Review Comment:
Yes, we need it to determine if this is a remote database which does not
support table pattern yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626312854
##
ql/src/test/results/clientpositive/llap/authorization_privilege_objects.q.out:
##
@@ -42,8 +42,8 @@ PREHOOK: query: SHOW TABLES IN test_auth_obj_db
PREHOOK: type: SHOWTABLES
PREHOOK: Input: database:test_auth_obj_db
filterListCmdObjects
-HIVE PRIVILEGE OBJECT { objectName: test_privs type: TABLE_OR_VIEW actionType:
OTHER dbName: test_auth_obj_db OWNER: testuser OWNERTYPE: USER}
-HIVE PRIVILEGE OBJECT { objectName: test_privs2 type: TABLE_OR_VIEW
actionType: OTHER dbName: test_auth_obj_db OWNER: testuser2 OWNERTYPE: USER}
+HIVE PRIVILEGE OBJECT { objectName: test_privs type: TABLE_OR_VIEW actionType:
OTHER dbName: test_auth_obj_db}
+HIVE PRIVILEGE OBJECT { objectName: test_privs2 type: TABLE_OR_VIEW
actionType: OTHER dbName: test_auth_obj_db}
Review Comment:
since we turns to request only table names instead of table objects, so here
auth check will not need owner info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626310126
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java:
##
@@ -78,20 +75,22 @@ private void showTables() throws HiveException {
}
private void showTablesExtended() throws HiveException {
-List tableObjects = new ArrayList<>();
-tableObjects.addAll(context.getDb().getTableObjects(desc.getDbName(),
null, desc.getTypeFilter()));
-if (desc.getPattern() != null) {
- Pattern pattern =
Pattern.compile(UDFLike.likePatternToRegExp(desc.getPattern()),
Pattern.CASE_INSENSITIVE);
- tableObjects = tableObjects.stream()
- .filter(object -> pattern.matcher(object.getTableName()).matches())
- .collect(Collectors.toList());
+List> tableTypePairs = new ArrayList<>();
+String pattern =
MetaStoreUtils.convertSqlPatternToRegExp(desc.getPattern());
+TableType typeFilter = desc.getTypeFilter();
+TableType[] tableTypes = typeFilter == null ? TableType.values() : new
TableType[]{typeFilter};
Review Comment:
It means to request all types table names if not input type filter in
statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27498: Support custom delimiter in SkippingTextInputFormat [hive]
mayankkunwar commented on code in PR #5265: URL: https://github.com/apache/hive/pull/5265#discussion_r1626308785 ## ql/src/test/results/clientpositive/llap/file_with_delimiter.q.out: ## @@ -0,0 +1,63 @@ +PREHOOK: query: create external table test(code string,name string) +ROW FORMAT SERDE + 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' + WITH SERDEPROPERTIES ( + 'field.delim'='\t') + STORED AS INPUTFORMAT + 'org.apache.hadoop.mapred.TextInputFormat' + OUTPUTFORMAT + 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' + A masked pattern was here + TBLPROPERTIES ( + 'skip.header.line.count'='1', + 'textinputformat.record.delimiter'='|') +PREHOOK: type: CREATETABLE + A masked pattern was here +PREHOOK: Output: database:default +PREHOOK: Output: default@test +POSTHOOK: query: create external table test(code string,name string) +ROW FORMAT SERDE + 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' + WITH SERDEPROPERTIES ( + 'field.delim'='\t') + STORED AS INPUTFORMAT + 'org.apache.hadoop.mapred.TextInputFormat' + OUTPUTFORMAT + 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' + A masked pattern was here + TBLPROPERTIES ( + 'skip.header.line.count'='1', + 'textinputformat.record.delimiter'='|') +POSTHOOK: type: CREATETABLE + A masked pattern was here +POSTHOOK: Output: database:default +POSTHOOK: Output: default@test +PREHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/header_footer_table_4/0003.txt' INTO TABLE test +PREHOOK: type: LOAD + A masked pattern was here +PREHOOK: Output: default@test +POSTHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/header_footer_table_4/0003.txt' INTO TABLE test +POSTHOOK: type: LOAD + A masked pattern was here +POSTHOOK: Output: default@test +PREHOOK: query: Select count(*) from test +PREHOOK: type: QUERY +PREHOOK: Input: default@test + A masked pattern was here +POSTHOOK: query: Select count(*) from test +POSTHOOK: type: QUERY +POSTHOOK: Input: default@test + A masked pattern was here +3 +PREHOOK: query: Select * from test +PREHOOK: type: QUERY +PREHOOK: Input: default@test + A masked pattern was here +POSTHOOK: query: Select * from test +POSTHOOK: type: QUERY +POSTHOOK: Input: default@test + A masked pattern was here +A NULL Review Comment: Ahh! got it. But the tab key in keyboard gives 4spaces in intellij text editor (which is not a real tab character). So the tab character(ASCII 009) can only be copy-pasted in the text files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27498: Support custom delimiter in SkippingTextInputFormat [hive]
mayankkunwar commented on code in PR #5265: URL: https://github.com/apache/hive/pull/5265#discussion_r1626308785 ## ql/src/test/results/clientpositive/llap/file_with_delimiter.q.out: ## @@ -0,0 +1,63 @@ +PREHOOK: query: create external table test(code string,name string) +ROW FORMAT SERDE + 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' + WITH SERDEPROPERTIES ( + 'field.delim'='\t') + STORED AS INPUTFORMAT + 'org.apache.hadoop.mapred.TextInputFormat' + OUTPUTFORMAT + 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' + A masked pattern was here + TBLPROPERTIES ( + 'skip.header.line.count'='1', + 'textinputformat.record.delimiter'='|') +PREHOOK: type: CREATETABLE + A masked pattern was here +PREHOOK: Output: database:default +PREHOOK: Output: default@test +POSTHOOK: query: create external table test(code string,name string) +ROW FORMAT SERDE + 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' + WITH SERDEPROPERTIES ( + 'field.delim'='\t') + STORED AS INPUTFORMAT + 'org.apache.hadoop.mapred.TextInputFormat' + OUTPUTFORMAT + 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' + A masked pattern was here + TBLPROPERTIES ( + 'skip.header.line.count'='1', + 'textinputformat.record.delimiter'='|') +POSTHOOK: type: CREATETABLE + A masked pattern was here +POSTHOOK: Output: database:default +POSTHOOK: Output: default@test +PREHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/header_footer_table_4/0003.txt' INTO TABLE test +PREHOOK: type: LOAD + A masked pattern was here +PREHOOK: Output: default@test +POSTHOOK: query: LOAD DATA LOCAL INPATH '../../data/files/header_footer_table_4/0003.txt' INTO TABLE test +POSTHOOK: type: LOAD + A masked pattern was here +POSTHOOK: Output: default@test +PREHOOK: query: Select count(*) from test +PREHOOK: type: QUERY +PREHOOK: Input: default@test + A masked pattern was here +POSTHOOK: query: Select count(*) from test +POSTHOOK: type: QUERY +POSTHOOK: Input: default@test + A masked pattern was here +3 +PREHOOK: query: Select * from test +PREHOOK: type: QUERY +PREHOOK: Input: default@test + A masked pattern was here +POSTHOOK: query: Select * from test +POSTHOOK: type: QUERY +POSTHOOK: Input: default@test + A masked pattern was here +A NULL Review Comment: Ahh! got it. But the tab key in keyboard gives 4spaces in intellij text editor. So the tab character(ASCII 009) can only be copy-pasted in the text files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-27498: Support custom delimiter in SkippingTextInputFormat [hive]
sonarcloud[bot] commented on PR #5265: URL: https://github.com/apache/hive/pull/5265#issuecomment-2147943643 ## [](https://sonarcloud.io/dashboard?id=apache_hive=5265) **Quality Gate passed** Issues  [28 New issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=false=true)  [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive=5265=WONTFIX) Measures  [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=5265=false=true)  No data about Coverage  No data about Duplication [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive=5265) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626287681
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesFormatter.java:
##
@@ -51,7 +52,7 @@ public static ShowTablesFormatter getFormatter(HiveConf conf)
{
public abstract void showTables(DataOutputStream out, List tables)
throws HiveException;
- abstract void showTablesExtended(DataOutputStream out, List tables)
throws HiveException;
+ abstract void showTablesExtended(DataOutputStream out, List> tables) throws HiveException;
Review Comment:
`SHOW TABLES EXTENDED` only outputs table name and table type, so we do not
need the `Table` objects here.
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##
@@ -1315,4 +1316,35 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ public static String convertSqlPatternToRegExp(String likePattern) {
Review Comment:
Yes, but `UDFLike.likePatternToRegExp` is used for
`java.util.regex.Pattern`, so it quote the characters which is not expected in
jdo String matches.
https://github.com/apache/hive/blob/e570e65a397aae47c62472f12acaaed06b68cf54/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java#L77-L82
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
