[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5151: Fix the SQL group-by for empty data table
fx19880617 commented on a change in pull request #5151: Fix the SQL group-by
for empty data table
URL: https://github.com/apache/incubator-pinot/pull/5151#discussion_r392559176
##
File path:
pinot-core/src/test/java/org/apache/pinot/core/common/datatable/DataTableUtilsTest.java
##
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.common.datatable;
+
+import java.io.IOException;
+import java.util.Collections;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.utils.CommonConstants.Broker.Request;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.DataTable;
+import org.apache.pinot.pql.parsers.Pql2Compiler;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+
+
+public class DataTableUtilsTest {
+ private static final Pql2Compiler COMPILER = new Pql2Compiler();
+
+ @Test
+ public void testBuildEmptyDataTable()
+ throws IOException {
+// Selection
+BrokerRequest brokerRequest = COMPILER.compileToBrokerRequest("SELECT *
FROM table WHERE foo = 'bar'");
+DataTable dataTable = DataTableUtils.buildEmptyDataTable(brokerRequest);
+DataSchema dataSchema = dataTable.getDataSchema();
+assertEquals(dataSchema.getColumnNames(), new String[]{"*"});
+assertEquals(dataSchema.getColumnDataTypes(), new
DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.STRING});
+assertEquals(dataTable.getNumberOfRows(), 0);
+
+// Aggregation
+brokerRequest = COMPILER.compileToBrokerRequest("SELECT COUNT(*), SUM(a),
MAX(b) FROM table WHERE foo = 'bar'");
+dataTable = DataTableUtils.buildEmptyDataTable(brokerRequest);
+dataSchema = dataTable.getDataSchema();
+assertEquals(dataSchema.getColumnNames(), new String[]{"count_star",
"sum_a", "max_b"});
+assertEquals(dataSchema.getColumnDataTypes(),
+new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.LONG,
DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE});
+assertEquals(dataTable.getNumberOfRows(), 1);
+assertEquals(dataTable.getLong(0, 0), 0L);
+assertEquals(dataTable.getDouble(0, 1), 0.0);
+assertEquals(dataTable.getDouble(0, 2), Double.NEGATIVE_INFINITY);
+
+// PQL group-by
+brokerRequest =
+COMPILER.compileToBrokerRequest("SELECT COUNT(*), SUM(a), MAX(b) FROM
table WHERE foo = 'bar' GROUP BY c, d");
+dataTable = DataTableUtils.buildEmptyDataTable(brokerRequest);
+dataSchema = dataTable.getDataSchema();
+assertEquals(dataSchema.getColumnNames(), new String[]{"functionName",
"GroupByResultMap"});
+assertEquals(dataSchema.getColumnDataTypes(),
+new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.STRING,
DataSchema.ColumnDataType.OBJECT});
+assertEquals(dataTable.getNumberOfRows(), 3);
+assertEquals(dataTable.getString(0, 0), "count_star");
+assertEquals(dataTable.getObject(0, 1), Collections.emptyMap());
+assertEquals(dataTable.getString(1, 0), "sum_a");
+assertEquals(dataTable.getObject(1, 1), Collections.emptyMap());
+assertEquals(dataTable.getString(2, 0), "max_b");
+assertEquals(dataTable.getObject(2, 1), Collections.emptyMap());
+
+// SQL group-by
+brokerRequest = COMPILER
+.compileToBrokerRequest("SELECT c, d, COUNT(*), SUM(a), MAX(b) FROM
table WHERE foo = 'bar' GROUP BY c, d");
+
brokerRequest.setQueryOptions(Collections.singletonMap(Request.QueryOptionKey.GROUP_BY_MODE,
Request.SQL));
+dataTable = DataTableUtils.buildEmptyDataTable(brokerRequest);
+dataSchema = dataTable.getDataSchema();
+assertEquals(dataSchema.getColumnNames(), new String[]{"c", "d",
"count_star", "sum_a", "max_b"});
Review comment:
new String[]{"c", "d", "count(*)", "sum(a)", "max(b)"}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5151: Fix the SQL group-by for empty data table
fx19880617 commented on a change in pull request #5151: Fix the SQL group-by
for empty data table
URL: https://github.com/apache/incubator-pinot/pull/5151#discussion_r392559176
##
File path:
pinot-core/src/test/java/org/apache/pinot/core/common/datatable/DataTableUtilsTest.java
##
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.common.datatable;
+
+import java.io.IOException;
+import java.util.Collections;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.utils.CommonConstants.Broker.Request;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.DataTable;
+import org.apache.pinot.pql.parsers.Pql2Compiler;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+
+
+public class DataTableUtilsTest {
+ private static final Pql2Compiler COMPILER = new Pql2Compiler();
+
+ @Test
+ public void testBuildEmptyDataTable()
+ throws IOException {
+// Selection
+BrokerRequest brokerRequest = COMPILER.compileToBrokerRequest("SELECT *
FROM table WHERE foo = 'bar'");
+DataTable dataTable = DataTableUtils.buildEmptyDataTable(brokerRequest);
+DataSchema dataSchema = dataTable.getDataSchema();
+assertEquals(dataSchema.getColumnNames(), new String[]{"*"});
+assertEquals(dataSchema.getColumnDataTypes(), new
DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.STRING});
+assertEquals(dataTable.getNumberOfRows(), 0);
+
+// Aggregation
+brokerRequest = COMPILER.compileToBrokerRequest("SELECT COUNT(*), SUM(a),
MAX(b) FROM table WHERE foo = 'bar'");
+dataTable = DataTableUtils.buildEmptyDataTable(brokerRequest);
+dataSchema = dataTable.getDataSchema();
+assertEquals(dataSchema.getColumnNames(), new String[]{"count_star",
"sum_a", "max_b"});
+assertEquals(dataSchema.getColumnDataTypes(),
+new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.LONG,
DataSchema.ColumnDataType.DOUBLE, DataSchema.ColumnDataType.DOUBLE});
+assertEquals(dataTable.getNumberOfRows(), 1);
+assertEquals(dataTable.getLong(0, 0), 0L);
+assertEquals(dataTable.getDouble(0, 1), 0.0);
+assertEquals(dataTable.getDouble(0, 2), Double.NEGATIVE_INFINITY);
+
+// PQL group-by
+brokerRequest =
+COMPILER.compileToBrokerRequest("SELECT COUNT(*), SUM(a), MAX(b) FROM
table WHERE foo = 'bar' GROUP BY c, d");
+dataTable = DataTableUtils.buildEmptyDataTable(brokerRequest);
+dataSchema = dataTable.getDataSchema();
+assertEquals(dataSchema.getColumnNames(), new String[]{"functionName",
"GroupByResultMap"});
+assertEquals(dataSchema.getColumnDataTypes(),
+new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.STRING,
DataSchema.ColumnDataType.OBJECT});
+assertEquals(dataTable.getNumberOfRows(), 3);
+assertEquals(dataTable.getString(0, 0), "count_star");
+assertEquals(dataTable.getObject(0, 1), Collections.emptyMap());
+assertEquals(dataTable.getString(1, 0), "sum_a");
+assertEquals(dataTable.getObject(1, 1), Collections.emptyMap());
+assertEquals(dataTable.getString(2, 0), "max_b");
+assertEquals(dataTable.getObject(2, 1), Collections.emptyMap());
+
+// SQL group-by
+brokerRequest = COMPILER
+.compileToBrokerRequest("SELECT c, d, COUNT(*), SUM(a), MAX(b) FROM
table WHERE foo = 'bar' GROUP BY c, d");
+
brokerRequest.setQueryOptions(Collections.singletonMap(Request.QueryOptionKey.GROUP_BY_MODE,
Request.SQL));
+dataTable = DataTableUtils.buildEmptyDataTable(brokerRequest);
+dataSchema = dataTable.getDataSchema();
+assertEquals(dataSchema.getColumnNames(), new String[]{"c", "d",
"count_star", "sum_a", "max_b"});
Review comment:
`count(*)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-pinot] branch master updated (82cfbdd -> d15a91a)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 82cfbdd Fix a formatting bug in AggregationFunctionUtils (#5148) add d15a91a update pinot assembly scripts (#5146) No new revisions were added by this update. Summary of changes: pinot-distribution/pinot-assembly.xml | 88 +-- pinot-plugins/pom.xml | 1 - 2 files changed, 64 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 merged pull request #5146: Update pinot assembly scripts for release 0.3.0
fx19880617 merged pull request #5146: Update pinot assembly scripts for release 0.3.0 URL: https://github.com/apache/incubator-pinot/pull/5146 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #5146: Update pinot assembly scripts for release 0.3.0
fx19880617 commented on issue #5146: Update pinot assembly scripts for release 0.3.0 URL: https://github.com/apache/incubator-pinot/pull/5146#issuecomment-598998494 > Whenever we add the new plugin and make it as part of the distribution, we will need to add the line here. We can periodically double check on this as part of the release process (similar that we clean up license for every release). > > Can we update the release doc to include this? Otherwise, LGTM! Thank you for working on this. Yes, also for required for plugin developer, this is how they could package plugin into the distribution. I will add this into not only release process doc, but also dev guide. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #5153: different result on fasthll and distinctcounthll
fx19880617 commented on issue #5153: different result on fasthll and distinctcounthll URL: https://github.com/apache/incubator-pinot/issues/5153#issuecomment-598991898 DistinctCountHLL could only infer bytes column as a hyperloglog object. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #5153: different result on fasthll and distinctcounthll
fx19880617 commented on issue #5153: different result on fasthll and distinctcounthll URL: https://github.com/apache/incubator-pinot/issues/5153#issuecomment-598991778 FastHll will convert one string into a hyperloglog object, which may represent thousand unique values. DistinctCountHLL treats string as a value, not hyperloglog object, so it will return the approximation of how many unique hyperloglog serialized strings, the value should be close to your total number scanned . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Fix a formatting bug in AggregationFunctionUtils (#5148)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 82cfbdd Fix a formatting bug in AggregationFunctionUtils (#5148)
82cfbdd is described below
commit 82cfbdda87767587f80544ea4b01b65cae1b54e8
Author: Xiaotian (Jackie) Jiang <1751+jackie-ji...@users.noreply.github.com>
AuthorDate: Fri Mar 13 18:13:40 2020 -0700
Fix a formatting bug in AggregationFunctionUtils (#5148)
---
.../function/AggregationFunctionUtils.java | 11 ++--
.../function/AggregationFunctionUtilsTest.java | 66 ++
2 files changed, 72 insertions(+), 5 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionUtils.java
b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionUtils.java
index 00c534d..133867a 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionUtils.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionUtils.java
@@ -111,12 +111,13 @@ public class AggregationFunctionUtils {
public static String formatValue(Object value) {
if (value instanceof Double) {
- Double doubleValue = (Double) value;
+ double doubleValue = (double) value;
- // String.format is very expensive, so avoid it for whole numbers that
can fit in Long.
- // We simply append ".0" to long, in order to keep the existing
behavior.
- if (doubleValue <= Long.MAX_VALUE &&
DoubleMath.isMathematicalInteger(doubleValue)) {
-return doubleValue.longValue() + ".0";
+ // NOTE: String.format() is very expensive, so avoid it for whole
numbers that can fit in Long.
+ // We simply append ".0" to long, in order to keep the
existing behavior.
+ if (doubleValue <= Long.MAX_VALUE && doubleValue >= Long.MIN_VALUE &&
DoubleMath
+ .isMathematicalInteger(doubleValue)) {
+return (long) doubleValue + ".0";
} else {
return String.format(Locale.US, "%1.5f", doubleValue);
}
diff --git
a/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionUtilsTest.java
b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionUtilsTest.java
new file mode 100644
index 000..ff74df9
--- /dev/null
+++
b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/AggregationFunctionUtilsTest.java
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.aggregation.function;
+
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+
+
+public class AggregationFunctionUtilsTest {
+
+ @Test
+ public void testFormatValue() {
+double value = Long.MAX_VALUE;
+assertEquals(AggregationFunctionUtils.formatValue(value),
"9223372036854775807.0");
+
+value = Long.MIN_VALUE;
+assertEquals(AggregationFunctionUtils.formatValue(value),
"-9223372036854775808.0");
+
+value = 1e30;
+assertEquals(AggregationFunctionUtils.formatValue(value),
"100.0");
+
+value = -1e30;
+assertEquals(AggregationFunctionUtils.formatValue(value),
"-100.0");
+
+value = 1e-3;
+assertEquals(AggregationFunctionUtils.formatValue(value), "0.00100");
+
+value = -1e-3;
+assertEquals(AggregationFunctionUtils.formatValue(value), "-0.00100");
+
+value = 1e-10;
+assertEquals(AggregationFunctionUtils.formatValue(value), "0.0");
+
+value = -1e-10;
+assertEquals(AggregationFunctionUtils.formatValue(value), "-0.0");
+
+value = 123.456789;
+assertEquals(AggregationFunctionUtils.formatValue(value), "123.45679");
+
+value = Double.POSITIVE_INFINITY;
+assertEquals(AggregationFunctionUtils.formatValue(value), "Infinity");
+
+value = Double.NEGATIVE_INFINITY;
+assertEquals(AggregationFunction
[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #5148: Fix a formatting bug in AggregationFunctionUtils
Jackie-Jiang merged pull request #5148: Fix a formatting bug in AggregationFunctionUtils URL: https://github.com/apache/incubator-pinot/pull/5148 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] xuanzih opened a new issue #5153: different result on fasthll and distinctcounthll
xuanzih opened a new issue #5153: different result on fasthll and
distinctcounthll
URL: https://github.com/apache/incubator-pinot/issues/5153
hi guys, we are trying to switch from fasthll to distinctcounthll.
`com.clearspring.analytics.stream.cardinality.HyperLogLog;` is used in code
and `org.apache.pinot.core.startree.hll.HllUtil` to serialize the hll to a
string.
with the same condition we have 1000x difference.
Example:
```
SELECT fasthll(my_hll), distinctcounthll(my_hll)
FROM counts_table WHERE timestamp >= 1500768000
```
I get results:
```
"aggregationResults": [
{
"function": "fastHLL_my_hll",
"value": "68685244"
}, {
"function": "distinctCountHLL_my_hll",
"value": "50535"
}]
```
Could anyone suggest what's the big difference between them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request #5142: [TE] Detection creation endpoints should trigger Replay & Tuning
xiaohui-sun commented on a change in pull request #5142: [TE] Detection creation endpoints should trigger Replay & Tuning URL: https://github.com/apache/incubator-pinot/pull/5142#discussion_r392539891 ## File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/YamlResource.java ## @@ -380,6 +380,9 @@ long createDetectionConfig(@NotNull String payload, long startTime, long endTime Long id = this.detectionConfigDAO.save(detectionConfig); Preconditions.checkNotNull(id, "Error while saving the detection pipeline"); +// create an yaml onboarding task to run replay and tuning +createYamlOnboardingTask(detectionConfig.getId(), startTime, endTime); Review comment: Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] snleee commented on issue #5146: Update pinot assembly scripts for release 0.3.0
snleee commented on issue #5146: Update pinot assembly scripts for release 0.3.0 URL: https://github.com/apache/incubator-pinot/pull/5146#issuecomment-598981186 Whenever we add the new plugin and make it as part of the distribution, we will need to add the line here. We can periodically double check on this as part of the release process (similar that we clean up license for every release). Can we update the release doc to include this? Otherwise, LGTM! Thank you for working on this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai opened a new pull request #5152: [TE][subscription] update subscription watermarks to use anomaly create time instead of end time
akshayrai opened a new pull request #5152: [TE][subscription] update subscription watermarks to use anomaly create time instead of end time URL: https://github.com/apache/incubator-pinot/pull/5152 Problem Statement: The current subscription watermarks are designed to notify an anomaly only once (even if merged) and we maintain this by keeping track of the last notified anomaly end time (watermark). However, the assumption here was that newer anomalies will always be detected on newer data (that is, newer anomalies can never have start time < watermark). This puts the restriction when dealing with backfilled data and also in the case of missing data where the actual deviation anomalies on the data might be detected later. This PR tries to remove this restriction by leveraging the anomaly create time in the watermark. Proposed changes: * Replace the anomaly end time with the anomaly create time in the vector clock. * Remove highWaterMark field from subscription config - As of today, we maintain 2 watermarks, namely the last notified anomaly ID(highWaterMark) and the anomaly end time watermark(vector clocks) to ensure that each anomaly is notified only once. The main purpose of the highWaterMark is to filter out merged anomalies from the time window. This is no longer required once we start relying on the anomaly create time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r392520612
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
Review comment:
I don't see a good way to test. I've been testing this by hooking this up to
the live ADLS Gen2. One way is to mock every single Azure SDK API that i'm
calling using Mockhito but this doesn't really check much.
Another potential approach is to create the integration test by
incorporating Azurite https://github.com/Azure/Azurite, which is Azure storage
service emulator. But, this doesn't support Azure Datalake Gen2.
By the way, I did verify all the functions by hooking up the live Data Lake
Gen2.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r392520612
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
Review comment:
I don't see a good way to test. I've been testing this by hooking this up to
the live ADLS Gen2. One way is to mock every single Azure SDK API that i'm
calling using Mockhito but this doesn't really check much.
Another potential approach is to create the integration test by
incorporating Azurite https://github.com/Azure/Azurite, which is Azure storage
service emulator. But, this doesn't support Azure Datalake Gen2.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r392519581
##
File path: pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/PinotFS.java
##
@@ -103,8 +106,13 @@ public boolean move(URI srcUri, URI dstUri, boolean
overwrite)
}
} else {
// ensures the parent path of dst exists.
- URI parentUri = Paths.get(dstUri).getParent().toUri();
- mkdir(parentUri);
+ try {
+Path parentPath = Paths.get(dstUri.getPath()).getParent();
Review comment:
added a test to cover this case to `LocalPinotFSTest`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5151: Fix the SQL group-by for empty data table
Jackie-Jiang opened a new pull request #5151: Fix the SQL group-by for empty data table URL: https://github.com/apache/incubator-pinot/pull/5151 When all segments are pruned on the server side, support SQL group-by empty data table. Enable SQL query tests for offline test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r392514077
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(AzureGen2PinotFS.class);
+
+ private static final String ACCOUNT_NAME = "accountName";
+ private static final String ACCESS_KEY = "accessKey";
+ private static final String FILE_SYSTEM_NAME = "fileSystemName";
+ private static final String ENABLE_CHECKSUM = "enableChecksum";
+
+ private static final String HTTPS_URL_PREFIX = "https://";;
+
+ private static final String AZURE_STORAGE_DNS_SUFFIX =
".dfs.core.windows.net";
+ private static final String AZURE_BLOB_DNS_SUFFIX = ".blob.core.windows.net";
+ private static final String PATH_ALREADY_EXISTS_ERROR_CODE =
"PathAlreadyExists";
+ private static final String IS_DIRECTORY_KEY = "hdi_isfolder";
+
+ private static final int NOT_FOUND_STATUS_CODE = 404;
+ private static final int ALREADY_EXISTS_STATUS_CODE = 409;
+
+ // Azure Data Lake Gen2's block size is 4MB
+ private static final int BUFFER_SIZE = 4 * 1024 * 1024;
+
+ private DataLakeFileSystemClient _fileSystemClient;
+ private BlobServiceClient _blobServiceClient;
+
+ // If enabled, pinotFS implementation will guarantee that the bits you've
read are the same as the ones you wrote.
+ // However, there's some overhead in computing hash. (Adds roughly 3 seconds
for 1GB file)
+ private boolean _enableChecksum;
+
+ @Override
+ public void init(Configuration config) {
+_enableChecksum = config.getBoolean(ENABLE_CHECKSUM, false);
+
+// Azure storage account name
+String accountName = config.getString(ACCOUNT_NAME);
+
+// TODO: consider to add the encryption of the following config
+String accessKey = config.getString(ACCESS_KEY);
+String fileSystemName = config.getString(FILE_SYSTEM_NAME);
+
+String dfsServiceEndpointUrl = HTTPS_URL_PREFIX + accountName +
AZURE_STORAGE
[GitHub] [incubator-pinot] snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r392512787
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(AzureGen2PinotFS.class);
+
+ private static final String ACCOUNT_NAME = "accountName";
+ private static final String ACCESS_KEY = "accessKey";
+ private static final String FILE_SYSTEM_NAME = "fileSystemName";
+ private static final String ENABLE_CHECKSUM = "enableChecksum";
+
+ private static final String HTTPS_URL_PREFIX = "https://";;
+
+ private static final String AZURE_STORAGE_DNS_SUFFIX =
".dfs.core.windows.net";
+ private static final String AZURE_BLOB_DNS_SUFFIX = ".blob.core.windows.net";
+ private static final String PATH_ALREADY_EXISTS_ERROR_CODE =
"PathAlreadyExists";
+ private static final String IS_DIRECTORY_KEY = "hdi_isfolder";
+
+ private static final int NOT_FOUND_STATUS_CODE = 404;
+ private static final int ALREADY_EXISTS_STATUS_CODE = 409;
+
+ // Azure Data Lake Gen2's block size is 4MB
+ private static final int BUFFER_SIZE = 4 * 1024 * 1024;
+
+ private DataLakeFileSystemClient _fileSystemClient;
+ private BlobServiceClient _blobServiceClient;
+
+ // If enabled, pinotFS implementation will guarantee that the bits you've
read are the same as the ones you wrote.
+ // However, there's some overhead in computing hash. (Adds roughly 3 seconds
for 1GB file)
+ private boolean _enableChecksum;
+
+ @Override
+ public void init(Configuration config) {
+_enableChecksum = config.getBoolean(ENABLE_CHECKSUM, false);
+
+// Azure storage account name
+String accountName = config.getString(ACCOUNT_NAME);
+
+// TODO: consider to add the encryption of the following config
+String accessKey = config.getString(ACCESS_KEY);
+String fileSystemName = config.getString(FILE_SYSTEM_NAME);
+
+String dfsServiceEndpointUrl = HTTPS_URL_PREFIX + accountName +
AZURE_STORAGE_D
[GitHub] [incubator-pinot] snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r392512787
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(AzureGen2PinotFS.class);
+
+ private static final String ACCOUNT_NAME = "accountName";
+ private static final String ACCESS_KEY = "accessKey";
+ private static final String FILE_SYSTEM_NAME = "fileSystemName";
+ private static final String ENABLE_CHECKSUM = "enableChecksum";
+
+ private static final String HTTPS_URL_PREFIX = "https://";;
+
+ private static final String AZURE_STORAGE_DNS_SUFFIX =
".dfs.core.windows.net";
+ private static final String AZURE_BLOB_DNS_SUFFIX = ".blob.core.windows.net";
+ private static final String PATH_ALREADY_EXISTS_ERROR_CODE =
"PathAlreadyExists";
+ private static final String IS_DIRECTORY_KEY = "hdi_isfolder";
+
+ private static final int NOT_FOUND_STATUS_CODE = 404;
+ private static final int ALREADY_EXISTS_STATUS_CODE = 409;
+
+ // Azure Data Lake Gen2's block size is 4MB
+ private static final int BUFFER_SIZE = 4 * 1024 * 1024;
+
+ private DataLakeFileSystemClient _fileSystemClient;
+ private BlobServiceClient _blobServiceClient;
+
+ // If enabled, pinotFS implementation will guarantee that the bits you've
read are the same as the ones you wrote.
+ // However, there's some overhead in computing hash. (Adds roughly 3 seconds
for 1GB file)
+ private boolean _enableChecksum;
+
+ @Override
+ public void init(Configuration config) {
+_enableChecksum = config.getBoolean(ENABLE_CHECKSUM, false);
+
+// Azure storage account name
+String accountName = config.getString(ACCOUNT_NAME);
+
+// TODO: consider to add the encryption of the following config
+String accessKey = config.getString(ACCESS_KEY);
+String fileSystemName = config.getString(FILE_SYSTEM_NAME);
+
+String dfsServiceEndpointUrl = HTTPS_URL_PREFIX + accountName +
AZURE_STORAGE_D
[GitHub] [incubator-pinot] snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r392512306
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(AzureGen2PinotFS.class);
+
+ private static final String ACCOUNT_NAME = "accountName";
+ private static final String ACCESS_KEY = "accessKey";
+ private static final String FILE_SYSTEM_NAME = "fileSystemName";
+ private static final String ENABLE_CHECKSUM = "enableChecksum";
+
+ private static final String HTTPS_URL_PREFIX = "https://";;
+
+ private static final String AZURE_STORAGE_DNS_SUFFIX =
".dfs.core.windows.net";
+ private static final String AZURE_BLOB_DNS_SUFFIX = ".blob.core.windows.net";
+ private static final String PATH_ALREADY_EXISTS_ERROR_CODE =
"PathAlreadyExists";
+ private static final String IS_DIRECTORY_KEY = "hdi_isfolder";
+
+ private static final int NOT_FOUND_STATUS_CODE = 404;
+ private static final int ALREADY_EXISTS_STATUS_CODE = 409;
+
+ // Azure Data Lake Gen2's block size is 4MB
+ private static final int BUFFER_SIZE = 4 * 1024 * 1024;
+
+ private DataLakeFileSystemClient _fileSystemClient;
+ private BlobServiceClient _blobServiceClient;
+
+ // If enabled, pinotFS implementation will guarantee that the bits you've
read are the same as the ones you wrote.
+ // However, there's some overhead in computing hash. (Adds roughly 3 seconds
for 1GB file)
+ private boolean _enableChecksum;
+
+ @Override
+ public void init(Configuration config) {
+_enableChecksum = config.getBoolean(ENABLE_CHECKSUM, false);
+
+// Azure storage account name
+String accountName = config.getString(ACCOUNT_NAME);
+
+// TODO: consider to add the encryption of the following config
+String accessKey = config.getString(ACCESS_KEY);
+String fileSystemName = config.getString(FILE_SYSTEM_NAME);
+
+String dfsServiceEndpointUrl = HTTPS_URL_PREFIX + accountName +
AZURE_STORAGE_D
[GitHub] [incubator-pinot] jackjlli opened a new issue #5150: Increase code coverage
jackjlli opened a new issue #5150: Increase code coverage URL: https://github.com/apache/incubator-pinot/issues/5150 Currently the code coverage of pinot repo is just 50% ([Link](https://codecov.io/github/apache/incubator-pinot)). This issue tracks the status of increasing code coverage. The following modules currently have less than 50% code coverage or don't have any test at all: 1. pinot-clients/pinot-java-client (46.15%) 2. pinot-common (47.96%) 3. pinot-plugins (36.38%) No code coverage: - pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-common - pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop - pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark - pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-stadardalone - pinot-plugins/pinot-file-system/pinot-gcs - pinot-plugins/pinot-file-system/pinot-hdfs The rest of the modules shown below seems fine but we should still bring them to at least 75%. pinot-broker (76.6%) pinot-controller (63.29%) pinot-core (76.85%) pinot-minion (68.98%) pinot-server (67.05%) pinot-spi (68.84%) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (6b6fa56a -> 888700d)
This is an automated email from the ASF dual-hosted git repository. akshayrai09 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 6b6fa56a Update license and notice (#5145) add 888700d [TE] Remove validation to make recipients backwards compatible (#5149) No new revisions were added by this update. Summary of changes: .../thirdeye/detection/validators/SubscriptionConfigValidator.java| 4 1 file changed, 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai merged pull request #5149: [TE] Remove validation to make recipients backwards compatible
akshayrai merged pull request #5149: [TE] Remove validation to make recipients backwards compatible URL: https://github.com/apache/incubator-pinot/pull/5149 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun commented on issue #5149: [TE] Remove validation to make recipients backwards compatible
xiaohui-sun commented on issue #5149: [TE] Remove validation to make recipients backwards compatible URL: https://github.com/apache/incubator-pinot/pull/5149#issuecomment-598814138 Thanks @akshayrai for the quick turnaround. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5142: [TE] Detection creation endpoints should trigger Replay & Tuning
akshayrai commented on a change in pull request #5142: [TE] Detection creation endpoints should trigger Replay & Tuning URL: https://github.com/apache/incubator-pinot/pull/5142#discussion_r392343210 ## File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/YamlResource.java ## @@ -380,6 +380,9 @@ long createDetectionConfig(@NotNull String payload, long startTime, long endTime Long id = this.detectionConfigDAO.save(detectionConfig); Preconditions.checkNotNull(id, "Error while saving the detection pipeline"); +// create an yaml onboarding task to run replay and tuning +createYamlOnboardingTask(detectionConfig.getId(), startTime, endTime); Review comment: @xiaohui-sun +1, I have filed a separate ticket to track the fuse mechanism. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai opened a new pull request #5149: [TE] Remove validation to make recipients backwards compatible
akshayrai opened a new pull request #5149: [TE] Remove validation to make recipients backwards compatible URL: https://github.com/apache/incubator-pinot/pull/5149 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
