[GitHub] [incubator-pinot] ly923976094 commented on issue #3978: The segments are stored in memory
ly923976094 commented on issue #3978: The segments are stored in memory URL: https://github.com/apache/incubator-pinot/issues/3978#issuecomment-474678713 > All segments of a table are divided among the tenant instances that host the table. If a server has a segment to serve, it will be loaded into memory. You can use rest commands on the controller to see the idealstate of a table, and that will indicate to you the servers that host each segment. > > It is up to you how you want to reduce cost. > > You can load the segments in "HEAP" mode, you just need to make sure that the maxDirectMemory argument to the jvm is sufficiently large so that the segments are loaded without OOM errors. > > You can load the segments in MMAP mode, in which case the segments are paged in from disk and Pinot lets the Operating System handle paging of segments in and out. If the load model changes from HEAP to MMAP, will the segments loaded into memory be released and then read data from the disk? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [TE] frontend - harleyjj/edit-alert - update endpoint for preview when editing alert (#3987)
This is an automated email from the ASF dual-hosted git repository.
jihao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 31f4fd0 [TE] frontend - harleyjj/edit-alert - update endpoint for
preview when editing alert (#3987)
31f4fd0 is described below
commit 31f4fd0f1f544baa4bc5af582fab0a053242c8bc
Author: Harley Jackson
AuthorDate: Tue Mar 19 17:35:52 2019 -0700
[TE] frontend - harleyjj/edit-alert - update endpoint for preview when
editing alert (#3987)
---
.../thirdeye-frontend/app/pods/components/alert-details/component.js | 2 +-
.../thirdeye-frontend/app/pods/components/yaml-editor/template.hbs| 1 +
thirdeye/thirdeye-frontend/app/utils/anomaly.js | 4 ++--
thirdeye/thirdeye-frontend/app/utils/api/anomaly.js | 4 ++--
4 files changed, 6 insertions(+), 5 deletions(-)
diff --git
a/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
b/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
index f93e432..37433f4 100644
--- a/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
+++ b/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
@@ -505,7 +505,7 @@ export default Component.extend({
let metricUrnList;
try {
if(isPreviewMode){
-applicationAnomalies = yield getYamlPreviewAnomalies(alertYaml, start,
end);
+applicationAnomalies = yield getYamlPreviewAnomalies(alertYaml, start,
end, alertId);
if (applicationAnomalies && applicationAnomalies.diagnostics &&
applicationAnomalies.diagnostics['0']) {
metricUrnList = Object.keys(applicationAnomalies.diagnostics['0']);
set(this, 'metricUrnList', metricUrnList);
diff --git
a/thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/template.hbs
b/thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/template.hbs
index 9aaae46..b2f01b0 100644
--- a/thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/template.hbs
+++ b/thirdeye/thirdeye-frontend/app/pods/components/yaml-editor/template.hbs
@@ -57,6 +57,7 @@
{{#alert-details
isPreviewMode=true
alertYaml=alertYaml
+alertId=alertId
dataIsCurrent=alertDataIsCurrent
}}
{{yield}}
diff --git a/thirdeye/thirdeye-frontend/app/utils/anomaly.js
b/thirdeye/thirdeye-frontend/app/utils/anomaly.js
index ed74925..3611aad 100644
--- a/thirdeye/thirdeye-frontend/app/utils/anomaly.js
+++ b/thirdeye/thirdeye-frontend/app/utils/anomaly.js
@@ -102,8 +102,8 @@ export function updateAnomalyFeedback(anomalyId,
feedbackType) {
* @param {Number} endTime - end time of analysis range
* @return {Ember.RSVP.Promise}
*/
-export function getYamlPreviewAnomalies(yamlString, startTime, endTime) {
- const url = getAnomaliesForYamlPreviewUrl(startTime, endTime);
+export function getYamlPreviewAnomalies(yamlString, startTime, endTime,
alertId) {
+ const url = getAnomaliesForYamlPreviewUrl(startTime, endTime, alertId);
return fetch(url, postYamlProps(yamlString)).then((res) => checkStatus(res,
'post', false, true));
}
diff --git a/thirdeye/thirdeye-frontend/app/utils/api/anomaly.js
b/thirdeye/thirdeye-frontend/app/utils/api/anomaly.js
index de4aa5a..9d5017e 100644
--- a/thirdeye/thirdeye-frontend/app/utils/api/anomaly.js
+++ b/thirdeye/thirdeye-frontend/app/utils/api/anomaly.js
@@ -16,8 +16,8 @@ export function getAnomalyDataUrl(anomalyId) {
* @returns {String} the complete yaml/preview url
* @example getAnomaliesForYamlPreview(150847270, 150847280) // yields
=> /yaml/preview?start=150847270=150847280=0=0
*/
-export function getAnomaliesForYamlPreviewUrl(startTime, endTime) {
- return
`/yaml/preview?start=${startTime}=${endTime}=0=0`;
+export function getAnomaliesForYamlPreviewUrl(startTime, endTime, alertId) {
+ return `/yaml/preview${alertId ? `/${alertId}` :
''}?start=${startTime}=${endTime}=0=0`;
}
/**
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh merged pull request #3987: [TE] frontend - harleyjj/edit-alert - update endpoint for preview whe…
jihaozh merged pull request #3987: [TE] frontend - harleyjj/edit-alert - update endpoint for preview whe… URL: https://github.com/apache/incubator-pinot/pull/3987 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun opened a new pull request #3990: [TE] Add algorithm based configurations into template
xiaohui-sun opened a new pull request #3990: [TE] Add algorithm based configurations into template URL: https://github.com/apache/incubator-pinot/pull/3990 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io commented on issue #3989: Add experiment section in getting started doc
codecov-io commented on issue #3989: Add experiment section in getting started doc URL: https://github.com/apache/incubator-pinot/pull/3989#issuecomment-474632607 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3989?src=pr=h1) Report > Merging [#3989](https://codecov.io/gh/apache/incubator-pinot/pull/3989?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/d2a3d84850386cd82d6a712a9be3ed5eb7146875?src=pr=desc) will **decrease** coverage by `0.12%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-pinot/pull/3989?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3989 +/- ## - Coverage 67.1% 66.97% -0.13% Complexity44 Files 1032 1032 Lines 5107051124 +54 Branches 7128 7135 +7 - Hits 3426934240 -29 - Misses1446114530 +69 - Partials 2340 2354 +14 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/3989?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | `75% <0%> (-25%)` | `0% <0%> (ø)` | | | [...egation/function/customobject/MinMaxRangePair.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9jdXN0b21vYmplY3QvTWluTWF4UmFuZ2VQYWlyLmphdmE=) | `75.86% <0%> (-24.14%)` | `0% <0%> (ø)` | | | [...r/validation/RealtimeSegmentValidationManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=) | `58.33% <0%> (-23.34%)` | `0% <0%> (ø)` | | | [...apache/pinot/common/metrics/ValidationMetrics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9WYWxpZGF0aW9uTWV0cmljcy5qYXZh) | `20.28% <0%> (-23.19%)` | `0% <0%> (ø)` | | | [.../impl/dictionary/LongOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh) | `81.81% <0%> (-10.91%)` | `0% <0%> (ø)` | | | [...pache/pinot/core/util/SortedRangeIntersection.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS91dGlsL1NvcnRlZFJhbmdlSW50ZXJzZWN0aW9uLmphdmE=) | `83.82% <0%> (-7.36%)` | `0% <0%> (ø)` | | | [...a/org/apache/pinot/common/config/TagNameUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1RhZ05hbWVVdGlscy5qYXZh) | `64.1% <0%> (-5.13%)` | `0% <0%> (ø)` | | | [...troller/helix/core/retention/RetentionManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JldGVudGlvbi9SZXRlbnRpb25NYW5hZ2VyLmphdmE=) | `75% <0%> (-4.17%)` | `0% <0%> (ø)` | | | [...ot/core/query/pruner/ColumnValueSegmentPruner.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9wcnVuZXIvQ29sdW1uVmFsdWVTZWdtZW50UHJ1bmVyLmphdmE=) | `87.32% <0%> (-2.82%)` | `0% <0%> (ø)` | | | [...org/apache/pinot/common/metrics/MetricsHelper.java](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NZXRyaWNzSGVscGVyLmphdmE=) | `72.5% <0%> (-2.5%)` | `0% <0%> (ø)` | | | ... and [23 more](https://codecov.io/gh/apache/incubator-pinot/pull/3989/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3989?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ =
[GitHub] [incubator-pinot] akshayrai merged pull request #3988: [TE] Remove deprecated legacy logic in user dashboard
akshayrai merged pull request #3988: [TE] Remove deprecated legacy logic in user dashboard URL: https://github.com/apache/incubator-pinot/pull/3988 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [TE] Remove deprecated legacy logic in user dashboard (#3988)
This is an automated email from the ASF dual-hosted git repository.
akshayrai09 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new f26b2f3 [TE] Remove deprecated legacy logic in user dashboard (#3988)
f26b2f3 is described below
commit f26b2f3d3855db351db2b9d3a7ec69de7c1a3b51
Author: Akshay Rai
AuthorDate: Tue Mar 19 16:40:47 2019 -0700
[TE] Remove deprecated legacy logic in user dashboard (#3988)
---
.../api/user/dashboard/UserDashboardResource.java | 140 +
.../dashboard/ThirdEyeDashboardApplication.java| 3 +-
.../resources/v2/pojo/AnomalySummary.java | 9 --
.../resource/v2/UserDashboardResourceTest.java | 97 +++---
4 files changed, 60 insertions(+), 189 deletions(-)
diff --git
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/user/dashboard/UserDashboardResource.java
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/user/dashboard/UserDashboardResource.java
index 871f34a..e2d7fc8 100644
---
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/user/dashboard/UserDashboardResource.java
+++
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/user/dashboard/UserDashboardResource.java
@@ -27,13 +27,8 @@ import com.wordnik.swagger.annotations.ApiParam;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
-import java.util.Comparator;
-import java.util.HashMap;
import java.util.HashSet;
-import java.util.Iterator;
import java.util.List;
-import java.util.Map;
-import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
@@ -46,18 +41,14 @@ import javax.ws.rs.core.MediaType;
import org.apache.commons.lang.StringUtils;
import org.apache.pinot.thirdeye.api.Constants;
import org.apache.pinot.thirdeye.constant.AnomalyFeedbackType;
-import org.apache.pinot.thirdeye.constant.AnomalyResultSource;
import org.apache.pinot.thirdeye.dashboard.resources.v2.ResourceUtils;
import org.apache.pinot.thirdeye.dashboard.resources.v2.pojo.AnomalySummary;
import org.apache.pinot.thirdeye.datalayer.bao.AlertConfigManager;
-import org.apache.pinot.thirdeye.datalayer.bao.AnomalyFunctionManager;
import org.apache.pinot.thirdeye.datalayer.bao.DatasetConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.DetectionAlertConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.DetectionConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.MergedAnomalyResultManager;
import org.apache.pinot.thirdeye.datalayer.bao.MetricConfigManager;
-import org.apache.pinot.thirdeye.datalayer.dto.AlertConfigDTO;
-import org.apache.pinot.thirdeye.datalayer.dto.AnomalyFunctionDTO;
import org.apache.pinot.thirdeye.datalayer.dto.DetectionAlertConfigDTO;
import org.apache.pinot.thirdeye.datalayer.dto.DetectionConfigDTO;
import org.apache.pinot.thirdeye.datalayer.dto.MergedAnomalyResultDTO;
@@ -65,14 +56,11 @@ import org.apache.pinot.thirdeye.datalayer.util.Predicate;
import org.apache.pinot.thirdeye.datasource.ThirdEyeCacheRegistry;
import org.apache.pinot.thirdeye.datasource.loader.AggregationLoader;
import org.apache.pinot.thirdeye.datasource.loader.DefaultAggregationLoader;
-import org.apache.pinot.thirdeye.detection.ConfigUtils;
import org.apache.pinot.thirdeye.detection.CurrentAndBaselineLoader;
import org.apache.pinot.thirdeye.rootcause.impl.MetricEntity;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import static
org.apache.pinot.thirdeye.detection.yaml.YamlDetectionAlertConfigTranslator.*;
-
/**
* Endpoints for user-customized dashboards (currently alerts only)
@@ -86,8 +74,6 @@ public class UserDashboardResource {
private static final int ANOMALIES_LIMIT_DEFAULT = 500;
private final MergedAnomalyResultManager anomalyDAO;
- private final AnomalyFunctionManager functionDAO;
- private final AlertConfigManager alertDAO;
private final MetricConfigManager metricDAO;
private final DatasetConfigManager datasetDAO;
private final DetectionConfigManager detectionDAO;
@@ -96,14 +82,12 @@ public class UserDashboardResource {
private final CurrentAndBaselineLoader currentAndBaselineLoader;
- public UserDashboardResource(MergedAnomalyResultManager anomalyDAO,
AnomalyFunctionManager functionDAO,
- MetricConfigManager metricDAO, DatasetConfigManager datasetDAO,
AlertConfigManager alertDAO,
- DetectionConfigManager detectionDAO, DetectionAlertConfigManager
detectionAlertDAO) {
+ public UserDashboardResource(MergedAnomalyResultManager anomalyDAO,
MetricConfigManager metricDAO,
+ DatasetConfigManager datasetDAO, DetectionConfigManager detectionDAO,
+ DetectionAlertConfigManager detectionAlertDAO) {
this.anomalyDAO = anomalyDAO;
-this.functionDAO = functionDAO;
this.metricDAO =
[incubator-pinot] branch master updated: Pinot server side change to optimize LLC segment completion with direct metadata upload. (#3941)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new fe203b5 Pinot server side change to optimize LLC segment completion
with direct metadata upload. (#3941)
fe203b5 is described below
commit fe203b5be2bbbf6464f1c757f07cb3374fa3bf19
Author: Ting Chen
AuthorDate: Tue Mar 19 16:35:41 2019 -0700
Pinot server side change to optimize LLC segment completion with direct
metadata upload. (#3941)
* The Pinot server side change to optimize LLC segment complete protocol by
uploading metadata directly.
* Minor style and comment fixes.
* Fix error handling and style issues.
* Add missing header..
* Add the config for commit end with metadata into server instance config.
Combine integration tests.
* Change comments.
* Minor comment fix.
* Use the metadata files created in the server directly.
* Deprecated the old segmentCommitEnd method and change the default to
commentEndWithMetadata.
* Return null instead of crashing server...
* Add comments.
* Invert config checks.
---
.../protocols/SegmentCompletionProtocol.java | 6 +++
.../apache/pinot/common/utils/CommonConstants.java | 1 +
.../common/utils/FileUploadDownloadClient.java | 24 +++
.../manager/config/InstanceDataManagerConfig.java | 2 +
.../realtime/LLRealtimeSegmentDataManager.java | 48 +++---
.../segment/index/loader/IndexLoadingConfig.java | 4 ++
.../ServerSegmentCompletionProtocolHandler.java| 37 +
.../realtime/LLRealtimeSegmentDataManagerTest.java | 4 +-
...CRealtimeClusterSplitCommitIntegrationTest.java | 2 +-
.../helix/HelixInstanceDataManagerConfig.java | 7
10 files changed, 126 insertions(+), 9 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java
b/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java
index 60ee1e0..822ad21 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java
@@ -359,6 +359,12 @@ public class SegmentCompletionProtocol {
}
}
+ public static class SegmentCommitEndWithMetadataRequest extends Request {
+public SegmentCommitEndWithMetadataRequest(Params params) {
+ super(params, MSG_TYPE_COMMIT_END_METADATA);
+}
+ }
+
public static class SegmentStoppedConsuming extends Request {
public SegmentStoppedConsuming(Params params) {
super(params, MSG_TYPE_STOPPED_CONSUMING);
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
index b76f2b8..0777ad4 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
@@ -156,6 +156,7 @@ public class CommonConstants {
public static final String CONFIG_OF_ENABLE_DEFAULT_COLUMNS =
"pinot.server.instance.enable.default.columns";
public static final String CONFIG_OF_ENABLE_SHUTDOWN_DELAY =
"pinot.server.instance.enable.shutdown.delay";
public static final String CONFIG_OF_ENABLE_SPLIT_COMMIT =
"pinot.server.instance.enable.split.commit";
+public static final String CONFIG_OF_ENABLE_COMMIT_END_WITH_METADATA =
"pinot.server.instance.enable.commitend.metadata";
public static final String CONFIG_OF_REALTIME_OFFHEAP_ALLOCATION =
"pinot.server.instance.realtime.alloc.offheap";
public static final String CONFIG_OF_REALTIME_OFFHEAP_DIRECT_ALLOCATION =
"pinot.server.instance.realtime.alloc.offheap.direct";
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java
index 4a37dcf..de8c76a 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java
@@ -29,6 +29,7 @@ import java.io.OutputStream;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.List;
+import java.util.Map;
import javax.annotation.Nullable;
import javax.net.ssl.SSLContext;
import org.apache.commons.io.IOUtils;
@@ -229,6 +230,22 @@ public class FileUploadDownloadClient implements Closeable
{
parameters, socketTimeoutMs);
}
+ private static HttpUriRequest getUploadSegmentMetadataFilesRequest(URI uri,
Map metadataFiles,
+ int segmentUploadRequestTimeoutMs) {
+
[GitHub] [incubator-pinot] mcvsubbu merged pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.
mcvsubbu merged pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload. URL: https://github.com/apache/incubator-pinot/pull/3941 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Add experiment section in getting started
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch add-doc-for-experiment
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 3d0a897efd9aecbe423dfd3eec41b9169398e2a7
Author: jackjlli
AuthorDate: Tue Mar 19 16:32:07 2019 -0700
Add experiment section in getting started
---
docs/getting_started.rst | 156 +++
1 file changed, 156 insertions(+)
diff --git a/docs/getting_started.rst b/docs/getting_started.rst
index e3d2416..774d3b3 100644
--- a/docs/getting_started.rst
+++ b/docs/getting_started.rst
@@ -94,3 +94,159 @@ show up in Pinot.
To show new events appearing, one can run :sql:`SELECT * FROM meetupRsvp ORDER
BY mtime DESC LIMIT 50` repeatedly, which shows the
last events that were ingested by Pinot.
+Experimenting with Pinot
+
+
+Now we have a quick start Pinot cluster running locally. The below shows a
step-by-step instruction on
+how to add a simple table to the Pinot system, how to upload segments, and how
to query it.
+
+Suppose we have a transcript in CSV format containing students' basic info and
their scores of each subject.
+
++++---+---+---+---+
+| studentID | firstName | lastName | gender | subject | score |
++++===+===+===+===+
+| 200| Lucy | Smith | Female | Maths |3.8|
++++---+---+---+---+
+| 200| Lucy | Smith | Female | English |3.5|
++++---+---+---+---+
+| 201| Bob|King |Male | Maths |3.2|
++++---+---+---+---+
+| 202| Nick | Young |Male | Physics |3.6|
++++---+---+---+---+
+
+Firstly in order to set up a table, we need to specify the schema of this
transcript.
+
+.. code-block:: none
+
+ {
+"schemaName": "transcript",
+"dimensionFieldSpecs": [
+ {
+"name": "studentID",
+"dataType": "STRING"
+ },
+ {
+"name": "firstName",
+"dataType": "STRING"
+ },
+ {
+"name": "lastName",
+"dataType": "STRING"
+ },
+ {
+"name": "gender",
+"dataType": "STRING"
+ },
+ {
+"name": "subject",
+"dataType": "STRING"
+ }
+],
+"metricFieldSpecs": [
+ {
+"name": "score",
+"dataType": "FLOAT"
+ }
+]
+ }
+
+To upload the schema, we can use the command below:
+
+.. code-block:: none
+
+ $
./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh
AddSchema -schemaFile /Users/jlli/transcript-schema.json -exec
+ Executing command: AddSchema -controllerHost 172.25.119.20 -controllerPort
9000 -schemaFilePath /Users/jlli/transcript-schema.json -exec
+ Sending request: http://172.25.119.20:9000/schemas to controller:
jlli-mn2.linkedin.biz, version:
0.1.0-SNAPSHOT-2c5d42a908213122ab0ad8b7ac9524fcf390e4cb
+
+Then, we need to specify the table config which links the schema to this table:
+
+.. code-block:: none
+
+ {
+"tableName": "transcript",
+"segmentsConfig" : {
+ "replication" : "1",
+ "schemaName" : "transcript",
+ "segmentAssignmentStrategy" : "BalanceNumSegmentAssignmentStrategy"
+},
+"tenants" : {
+ "broker":"DefaultTenant",
+ "server":"DefaultTenant"
+},
+"tableIndexConfig" : {
+ "invertedIndexColumns" : [],
+ "loadMode" : "HEAP",
+ "lazyLoad" : "false"
+},
+"tableType":"OFFLINE",
+"metadata": {}
+ }
+
+And upload the table config to Pinot cluster:
+
+.. code-block:: none
+
+ $
./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh
AddTable -filePath /Users/jlli/transcript-table-config.json -exec
+ Executing command: AddTable -filePath
/Users/jlli/transcript-table-config.json -controllerHost 172.25.119.20
-controllerPort 9000 -exec
+ {"status":"Table transcript_OFFLINE successfully added"}
+
+In order to upload our data to Pinot cluster, we need to convert our CSV file
to Pinot Segment:
+
+.. code-block:: none
+
+ $
./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh
CreateSegment -dataDir /Users/jlli/Desktop/test/ -format CSV -outDir
/Users/jlli/Desktop/test2/ -tableName transcript -segmentName transcript_0
-overwrite -schemaFile /Users/jlli/transcript-schema.json
+ Executing command: CreateSegment -generatorConfigFile null -dataDir
/Users/jlli/Desktop/test/ -format CSV -outDir
[GitHub] [incubator-pinot] jackjlli opened a new pull request #3989: Add experiment section in getting started doc
jackjlli opened a new pull request #3989: Add experiment section in getting started doc URL: https://github.com/apache/incubator-pinot/pull/3989 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch add-doc-for-experiment created (now 3d0a897)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch add-doc-for-experiment in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 3d0a897 Add experiment section in getting started This branch includes the following new commits: new 3d0a897 Add experiment section in getting started The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch add-doc-experiment-with-pinot updated (f456b46 -> dd11db8)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch add-doc-experiment-with-pinot in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard f456b46 [TE] frontend - harleyjj/report-anomaly - adds back report-anomaly modal to alert overview (#3985) new dd11db8 [TE] frontend - harleyjj/report-anomaly - adds back report-anomaly modal to alert overview (#3985) This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (f456b46) \ N -- N -- N refs/heads/add-doc-experiment-with-pinot (dd11db8) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 5853 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/getting_started.rst | 34 ++ 1 file changed, 26 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai opened a new pull request #3988: [TE] Remove deprecated legacy logic in user dashboard
akshayrai opened a new pull request #3988: [TE] Remove deprecated legacy logic in user dashboard URL: https://github.com/apache/incubator-pinot/pull/3988 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: [TE] frontend - harleyjj/report-anomaly - adds back report-anomaly modal to alert overview (#3985)
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch add-doc-experiment-with-pinot
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit f456b46186b719c2dbfcb43c77e924cfff36517e
Author: Harley Jackson
AuthorDate: Tue Mar 19 10:24:01 2019 -0700
[TE] frontend - harleyjj/report-anomaly - adds back report-anomaly modal to
alert overview (#3985)
---
docs/getting_started.rst | 138 +++
.../app/pods/components/alert-details/component.js | 146 +++--
.../app/pods/components/alert-details/template.hbs | 50 +++
.../components/alert-report-modal/component.js | 55
.../components/alert-report-modal/template.hbs | 61 -
.../app/pods/components/yaml-editor/component.js | 8 +-
.../app/pods/manage/explore/template.hbs | 1 +
thirdeye/thirdeye-frontend/app/utils/anomaly.js| 4 +-
.../thirdeye-frontend/app/utils/api/anomaly.js | 4 +-
.../tests/unit/utils/api/anomaly-test.js | 2 +-
10 files changed, 391 insertions(+), 78 deletions(-)
diff --git a/docs/getting_started.rst b/docs/getting_started.rst
index e3d2416..78a9250 100644
--- a/docs/getting_started.rst
+++ b/docs/getting_started.rst
@@ -94,3 +94,141 @@ show up in Pinot.
To show new events appearing, one can run :sql:`SELECT * FROM meetupRsvp ORDER
BY mtime DESC LIMIT 50` repeatedly, which shows the
last events that were ingested by Pinot.
+Experimenting with Pinot
+
+
+Now we have a quick start Pinot cluster running locally. The below shows a
step-by-step instruction on
+how to add a simple table to the Pinot system, how to upload segments, and how
to query it.
+
+Suppose we have a transcript in CSV format containing students' basic info and
their scores of each subject.
+
++++---+---+---+---+
+| StudentID | firstName | lastName | gender | subject | score |
++++===+===+===+===+
+| 200| Lucy | Smith | Female | Maths |3.8|
++++---+---+---+---+
+| 200| Lucy | Smith | Female | English |3.5|
++++---+---+---+---+
+| 201| Bob|King |Male | Maths |3.2|
++++---+---+---+---+
+| 202| Nick | Young |Male | Physics |3.6|
++++---+---+---+---+
+
+Firstly in order to set up a table, we need to specify the schema of this
transcript.
+
+.. code-block:: none
+
+ {
+"schemaName": "transcript",
+"dimensionFieldSpecs": [
+ {
+"name": "studentID",
+"dataType": "STRING"
+ },
+ {
+"name": "firstName",
+"dataType": "STRING"
+ },
+ {
+"name": "lastName",
+"dataType": "STRING"
+ },
+ {
+"name": "gender",
+"dataType": "STRING"
+ },
+ {
+"name": "subject",
+"dataType": "STRING"
+ }
+],
+"metricFieldSpecs": [
+ {
+"name": "score",
+"dataType": "INT"
+ }
+]
+ }
+
+To upload the schema, we can use the command below:
+
+.. code-block:: none
+
+ $
./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh
AddSchema -schemaFile /Users/jlli/transcript-schema.json -exec
+ Executing command: AddSchema -controllerHost 172.25.119.20 -controllerPort
9000 -schemaFilePath /Users/jlli/transcript-schema.json -exec
+ Sending request: http://172.25.119.20:9000/schemas to controller:
jlli-mn2.linkedin.biz, version:
0.1.0-SNAPSHOT-2c5d42a908213122ab0ad8b7ac9524fcf390e4cb
+
+Then, we need to specify the table config which links the schema to this table:
+
+.. code-block:: none
+
+ {
+"tableName": "transcript",
+"segmentsConfig" : {
+ "replication" : "1",
+ "schemaName" : "transcript",
+ "segmentAssignmentStrategy" : "BalanceNumSegmentAssignmentStrategy"
+},
+"tenants" : {
+ "broker":"DefaultTenant",
+ "server":"DefaultTenant"
+},
+"tableIndexConfig" : {
+ "invertedIndexColumns" : [],
+ "loadMode" : "HEAP",
+ "lazyLoad" : "false"
+},
+"tableType":"OFFLINE",
+"metadata": {}
+ }
+
+And upload the table config to Pinot cluster:
+
+.. code-block:: none
+
+ $
./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh
AddTable -filePath /Users/jlli/transcript-table-config.json -exec
+ Executing command: AddTable -filePath
/Users/jlli/transcript-table-config.json -controllerHost 172.25.119.20
[incubator-pinot] branch add-doc-experiment-with-pinot created (now f456b46)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch add-doc-experiment-with-pinot in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at f456b46 [TE] frontend - harleyjj/report-anomaly - adds back report-anomaly modal to alert overview (#3985) This branch includes the following new commits: new f456b46 [TE] frontend - harleyjj/report-anomaly - adds back report-anomaly modal to alert overview (#3985) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [TE] Fix for delayed anomalies due to watermark bug (#3984)
This is an automated email from the ASF dual-hosted git repository.
akshayrai09 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new d2a3d84 [TE] Fix for delayed anomalies due to watermark bug (#3984)
d2a3d84 is described below
commit d2a3d84850386cd82d6a712a9be3ed5eb7146875
Author: Akshay Rai
AuthorDate: Tue Mar 19 16:00:44 2019 -0700
[TE] Fix for delayed anomalies due to watermark bug (#3984)
---
.../apache/pinot/thirdeye/detection/DetectionUtils.java| 4 ++--
.../thirdeye/detection/algorithm/MergeWrapperTest.java | 14 +++---
.../detection/wrapper/ChildKeepingMergeWrapperTest.java| 12 ++--
3 files changed, 15 insertions(+), 15 deletions(-)
diff --git
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
index 50e4a98..f2ca286 100644
---
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
+++
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
@@ -168,10 +168,10 @@ public class DetectionUtils {
* @return the last time stamp
*/
public static long consolidateNestedLastTimeStamps(Collection
nestedLastTimeStamps){
-if(nestedLastTimeStamps.isEmpty() || nestedLastTimeStamps.contains(-1L)){
+if(nestedLastTimeStamps.isEmpty()){
return -1L;
}
-return Collections.min(nestedLastTimeStamps);
+return Collections.max(nestedLastTimeStamps);
}
/**
diff --git
a/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/MergeWrapperTest.java
b/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/MergeWrapperTest.java
index a32e76f..1f679a0 100644
---
a/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/MergeWrapperTest.java
+++
b/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/MergeWrapperTest.java
@@ -116,7 +116,7 @@ public class MergeWrapperTest {
DetectionPipelineResult output = this.wrapper.run();
Assert.assertEquals(output.getAnomalies().size(), 5);
-Assert.assertEquals(output.getLastTimestamp(), 2900);
+Assert.assertEquals(output.getLastTimestamp(), 3000);
}
@Test
@@ -127,7 +127,7 @@ public class MergeWrapperTest {
DetectionPipelineResult output = this.wrapper.run();
Assert.assertEquals(output.getAnomalies().size(), 3);
-Assert.assertEquals(output.getLastTimestamp(), 2900);
+Assert.assertEquals(output.getLastTimestamp(), 3000);
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(0, 1250)));
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(1500, 2000)));
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(2200, 2800)));
@@ -142,7 +142,7 @@ public class MergeWrapperTest {
DetectionPipelineResult output = this.wrapper.run();
Assert.assertEquals(output.getAnomalies().size(), 3);
-Assert.assertEquals(output.getLastTimestamp(), 2900);
+Assert.assertEquals(output.getLastTimestamp(), 3000);
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(0, 1250)));
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(1500, 2300)));
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(2400, 2800)));
@@ -168,7 +168,7 @@ public class MergeWrapperTest {
DetectionPipelineResult output = this.wrapper.run();
Assert.assertEquals(output.getAnomalies().size(), 4);
-Assert.assertEquals(output.getLastTimestamp(), 2900);
+Assert.assertEquals(output.getLastTimestamp(), 3700);
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(0, 1250)));
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(1500, 2300)));
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(2400, 3650)));
@@ -195,7 +195,7 @@ public class MergeWrapperTest {
DetectionPipelineResult output = this.wrapper.run();
Assert.assertEquals(output.getAnomalies().size(), 4);
-Assert.assertEquals(output.getLastTimestamp(), 2900);
+Assert.assertEquals(output.getLastTimestamp(), 3700);
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(0, 1250)));
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(1500, 2300)));
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(2400, 3650)));
@@ -221,7 +221,7 @@ public class MergeWrapperTest {
DetectionPipelineResult output = this.wrapper.run();
Assert.assertEquals(output.getAnomalies().size(), 8);
-Assert.assertEquals(output.getLastTimestamp(), 2900);
+Assert.assertEquals(output.getLastTimestamp(), 3700);
Assert.assertTrue(output.getAnomalies().contains(makeAnomaly(0, 500)));
[GitHub] [incubator-pinot] akshayrai merged pull request #3984: [TE] Fix for delayed anomalies due to watermark bug
akshayrai merged pull request #3984: [TE] Fix for delayed anomalies due to watermark bug URL: https://github.com/apache/incubator-pinot/pull/3984 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch orc updated (de8e3e8 -> eb5e528)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch orc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard de8e3e8 Using local path to read segment discard 4a3ce08 Adding orc reader new eb5e528 Adding orc reader This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (de8e3e8) \ N -- N -- N refs/heads/orc (eb5e528) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 5849 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../pinot/orc/data/readers/ORCRecordReader.java| 27 +- 1 file changed, 26 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch remotefiles deleted (was ef6b912)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch remotefiles in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was ef6b912 Changing segment generator config to not assume file abstraction for input file paths The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch orc updated: Using local path to read segment
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch orc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/orc by this push:
new de8e3e8 Using local path to read segment
de8e3e8 is described below
commit de8e3e8ea8e5c390d5668372341232d866dab361
Author: Jennifer Dai
AuthorDate: Tue Mar 19 13:59:49 2019 -0700
Using local path to read segment
---
.../main/java/org/apache/pinot/orc/data/readers/ORCRecordReader.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/pinot-orc/src/main/java/org/apache/pinot/orc/data/readers/ORCRecordReader.java
b/pinot-orc/src/main/java/org/apache/pinot/orc/data/readers/ORCRecordReader.java
index 68660fd..754be0f 100644
---
a/pinot-orc/src/main/java/org/apache/pinot/orc/data/readers/ORCRecordReader.java
+++
b/pinot-orc/src/main/java/org/apache/pinot/orc/data/readers/ORCRecordReader.java
@@ -49,7 +49,9 @@ public class ORCRecordReader implements RecordReader {
Configuration conf = new Configuration();
LOGGER.info("Creating segment for {}",
segmentGeneratorConfig.getInputFilePath());
try {
- _reader = OrcFile.createReader(new
Path(segmentGeneratorConfig.getInputFilePath()), OrcFile.readerOptions(conf));
+ Path orcReaderPath = new Path("file://" +
segmentGeneratorConfig.getInputFilePath());
+ LOGGER.info("orc reader path is {}", orcReaderPath);
+ _reader = OrcFile.createReader(orcReaderPath,
OrcFile.readerOptions(conf));
_orcSchema = _reader.getSchema();
LOGGER.info("ORC schema is {}", _orcSchema.toJson());
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment
codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment URL: https://github.com/apache/incubator-pinot/pull/3975#issuecomment-473083001 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=h1) Report > Merging [#3975](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/5867b16f32145a80fa98de4d24ec6302b17bb95e?src=pr=desc) will **increase** coverage by `0.04%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3975 +/- ## + Coverage 66.99% 67.03% +0.04% Complexity44 Files 1032 1032 Lines 5104751070 +23 Branches 7126 7128 +2 + Hits 3420034236 +36 + Misses1450214490 -12 + Partials 2345 2344 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | `75% <0%> (-25%)` | `0% <0%> (ø)` | | | [.../impl/dictionary/LongOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh) | `81.81% <0%> (-10.91%)` | `0% <0%> (ø)` | | | [...impl/dictionary/FloatOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRmxvYXRPZmZIZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `81.81% <0%> (-10.91%)` | `0% <0%> (ø)` | | | [...thandler/SingleConnectionBrokerRequestHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvU2luZ2xlQ29ubmVjdGlvbkJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `79.54% <0%> (-9.1%)` | `0% <0%> (ø)` | | | [...elix/core/relocation/RealtimeSegmentRelocator.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlbG9jYXRpb24vUmVhbHRpbWVTZWdtZW50UmVsb2NhdG9yLmphdmE=) | `73.62% <0%> (-7.7%)` | `0% <0%> (ø)` | | | [...n/java/org/apache/pinot/core/transport/Server.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS90cmFuc3BvcnQvU2VydmVyLmphdmE=) | `59.25% <0%> (-7.41%)` | `0% <0%> (ø)` | | | [.../apache/pinot/common/config/RealtimeTagConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1JlYWx0aW1lVGFnQ29uZmlnLmphdmE=) | `93.33% <0%> (-6.67%)` | `0% <0%> (ø)` | | | [...impl/dictionary/DoubleOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `68.88% <0%> (-6.67%)` | `0% <0%> (ø)` | | | [...a/org/apache/pinot/common/config/TagNameUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1RhZ05hbWVVdGlscy5qYXZh) | `64.1% <0%> (-5.13%)` | `0% <0%> (ø)` | | | [...g/apache/pinot/common/metrics/AbstractMetrics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9BYnN0cmFjdE1ldHJpY3MuamF2YQ==) | `75.33% <0%> (-1.34%)` | `0% <0%> (ø)` | | | ... and [24 more](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=continue). > **Legend** - [Click here to learn
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.
mcvsubbu commented on a change in pull request #3941: Pinot server side change
to optimize LLC segment completion with direct metadata upload.
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r267074696
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
##
@@ -735,7 +765,13 @@ protected SegmentBuildDescriptor
buildSegmentInternal(boolean forCommit) {
if (_isOffHeap) {
params.withMemoryUsedBytes(_memoryManager.getTotalAllocatedBytes());
}
-SegmentCompletionProtocol.Response commitEndResponse =
_protocolHandler.segmentCommitEnd(params);
+SegmentCompletionProtocol.Response commitEndResponse;
+if (!_indexLoadingConfig.isEnableSplitCommitEndWithMetadata()) {
Review comment:
nit I would prefer the condition inverted, given that both if and else
portions are one line long.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch getting-started-doc created (now 7cc4401)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch getting-started-doc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 7cc4401 Add doc on experimenting with Pinot This branch includes the following new commits: new 7cc4401 Add doc on experimenting with Pinot The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Add doc on experimenting with Pinot
This is an automated email from the ASF dual-hosted git repository. jlli pushed a commit to branch getting-started-doc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 7cc440124cf8c031cf0999f04c9625589be71f31 Author: jackjlli AuthorDate: Tue Mar 19 13:20:07 2019 -0700 Add doc on experimenting with Pinot --- docs/getting_started.rst | 7 +++ 1 file changed, 7 insertions(+) diff --git a/docs/getting_started.rst b/docs/getting_started.rst index e3d2416..4b92f96 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -94,3 +94,10 @@ show up in Pinot. To show new events appearing, one can run :sql:`SELECT * FROM meetupRsvp ORDER BY mtime DESC LIMIT 50` repeatedly, which shows the last events that were ingested by Pinot. +Experiment with Pinot +~ + +Here's the instruction on how to add a simple table to the pinot system you got working int the previous demo steps, +how to add segments, and how to query it. + + - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] harleyjj opened a new pull request #3987: [TE] frontend - harleyjj/edit-alert - update endpoint for preview whe…
harleyjj opened a new pull request #3987: [TE] frontend - harleyjj/edit-alert - update endpoint for preview whe… URL: https://github.com/apache/incubator-pinot/pull/3987 …n editing alert This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] chenboat commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.
chenboat commented on a change in pull request #3941: Pinot server side change
to optimize LLC segment completion with direct metadata upload.
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r267053088
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
##
@@ -681,11 +690,23 @@ protected SegmentBuildDescriptor
buildSegmentInternal(boolean forCommit) {
TimeUnit.MILLISECONDS.toSeconds(waitTimeMillis));
if (forCommit) {
+Preconditions.checkState(destDir.listFiles() != null &&
destDir.listFiles().length > 0,
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] chenboat commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.
chenboat commented on a change in pull request #3941: Pinot server side change
to optimize LLC segment completion with direct metadata upload.
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r267053012
##
File path:
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManagerConfig.java
##
@@ -163,6 +165,11 @@ public boolean isEnableSplitCommit() {
return _instanceDataManagerConfiguration.getBoolean(ENABLE_SPLIT_COMMIT,
false);
}
+ @Override
+ public boolean isEnableSplitCommitEndWithMetadata() {
+return
_instanceDataManagerConfiguration.getBoolean(ENABLE_SPLIT_COMMIT_END_WITH_METADATA,
true);
+ }
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment
jgutmann commented on a change in pull request #3975: ReadTheDocs documentation
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267040411
##
File path: docs/tableconfig_schema.rst
##
@@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings
in the table config which will influence how Pinot operates. The first and most
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically
uploaded - daily, weekly, etc. A realtime table, however, is used to consume
data from incoming data streams and serve this data in a near-realtime manner.
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections
will be explained and if applicable have appropriate sections linked to for
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'),
or underscores ('_'). Though using a double-underscore ('__') is not allowed
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or
``REALTIME``. There are some settings specific to each type. This
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+ "tableName": "myPinotTable",
+ "tableType": "REALTIME"
+ "segmentsConfig": {},
+ "tableIndexConfig": {},
+ "tenants": {},
+ "routing": {},
+ "task": {},
+ "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the ``retentionTimeUnit`` and
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and
``replicasPerPartition`` for realtime tables will indicate how many replicas of
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+ "retentionTimeUnit": "DAYS",
+ "retentionTimeValue": "5",
+ "segmentPushFrequency": "daily",
+ "segmentPushType": "APPEND",
+ "replication": "3",
+ "replicasPerPartition": "3",
+ "schemaName": "ugcGestureEvents",
+ "timeColumnName": "daysSinceEpoch",
+ "timeType": "DAYS",
+ "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list
of real column names as specified in the schema. Column names present will NOT
have a dictionary created. More info on indexes can be found on the `Index
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column
names as specified in the schema.
+* Aggregate Metrics - Using ``aggregateMetrics`` set to ``"true"`` to enable
the feature and ``"false"`` to disable. This feature is only available on
REALTIME tables.
Review comment:
I will add one sentence, but beyond that is beyond the
[GitHub] [incubator-pinot] jgutmann commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment
jgutmann commented on a change in pull request #3975: ReadTheDocs documentation
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r267035183
##
File path: docs/tableconfig_schema.rst
##
@@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings
in the table config which will influence how Pinot operates. The first and most
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically
uploaded - daily, weekly, etc. A realtime table, however, is used to consume
data from incoming data streams and serve this data in a near-realtime manner.
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections
will be explained and if applicable have appropriate sections linked to for
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'),
or underscores ('_'). Though using a double-underscore ('__') is not allowed
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or
``REALTIME``. There are some settings specific to each type. This
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+ "tableName": "myPinotTable",
+ "tableType": "REALTIME"
+ "segmentsConfig": {},
+ "tableIndexConfig": {},
+ "tenants": {},
+ "routing": {},
+ "task": {},
+ "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
Review comment:
configuring the list of items below it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io commented on issue #3986: Modify documentation in managing Pinot page
codecov-io commented on issue #3986: Modify documentation in managing Pinot page URL: https://github.com/apache/incubator-pinot/pull/3986#issuecomment-474510749 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3986?src=pr=h1) Report > Merging [#3986](https://codecov.io/gh/apache/incubator-pinot/pull/3986?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/2c5d42a908213122ab0ad8b7ac9524fcf390e4cb?src=pr=desc) will **decrease** coverage by `0.05%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-pinot/pull/3986?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#3986 +/- ## - Coverage 67.1% 67.04% -0.06% Complexity44 Files 1032 1032 Lines 5107051070 Branches 7128 7128 - Hits 3426934242 -27 - Misses1446114487 +26 - Partials 2340 2341 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/3986?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | `75% <0%> (-25%)` | `0% <0%> (ø)` | | | [...egation/function/customobject/MinMaxRangePair.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9jdXN0b21vYmplY3QvTWluTWF4UmFuZ2VQYWlyLmphdmE=) | `75.86% <0%> (-24.14%)` | `0% <0%> (ø)` | | | [...r/validation/RealtimeSegmentValidationManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=) | `58.33% <0%> (-23.34%)` | `0% <0%> (ø)` | | | [...apache/pinot/common/metrics/ValidationMetrics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9WYWxpZGF0aW9uTWV0cmljcy5qYXZh) | `20.28% <0%> (-23.19%)` | `0% <0%> (ø)` | | | [...pache/pinot/core/util/SortedRangeIntersection.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS91dGlsL1NvcnRlZFJhbmdlSW50ZXJzZWN0aW9uLmphdmE=) | `83.82% <0%> (-7.36%)` | `0% <0%> (ø)` | | | [.../impl/dictionary/LongOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh) | `87.27% <0%> (-5.46%)` | `0% <0%> (ø)` | | | [...a/org/apache/pinot/common/config/TagNameUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1RhZ05hbWVVdGlscy5qYXZh) | `64.1% <0%> (-5.13%)` | `0% <0%> (ø)` | | | [...ot/core/query/pruner/ColumnValueSegmentPruner.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9wcnVuZXIvQ29sdW1uVmFsdWVTZWdtZW50UHJ1bmVyLmphdmE=) | `87.32% <0%> (-2.82%)` | `0% <0%> (ø)` | | | [...org/apache/pinot/common/metrics/MetricsHelper.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NZXRyaWNzSGVscGVyLmphdmE=) | `72.5% <0%> (-2.5%)` | `0% <0%> (ø)` | | | [.../controller/helix/core/SegmentDeletionManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL1NlZ21lbnREZWxldGlvbk1hbmFnZXIuamF2YQ==) | `77.59% <0%> (-2.41%)` | `0% <0%> (ø)` | | | ... and [21 more](https://codecov.io/gh/apache/incubator-pinot/pull/3986/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3986?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute
[GitHub] [incubator-pinot] npawar commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.
npawar commented on a change in pull request #3941: Pinot server side change to
optimize LLC segment completion with direct metadata upload.
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r267031148
##
File path:
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManagerConfig.java
##
@@ -163,6 +165,11 @@ public boolean isEnableSplitCommit() {
return _instanceDataManagerConfiguration.getBoolean(ENABLE_SPLIT_COMMIT,
false);
}
+ @Override
+ public boolean isEnableSplitCommitEndWithMetadata() {
+return
_instanceDataManagerConfiguration.getBoolean(ENABLE_SPLIT_COMMIT_END_WITH_METADATA,
true);
+ }
Review comment:
I see Subbu's comments about making default true. But the PR description
says it will be default false. Please update the description in that case
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.
npawar commented on a change in pull request #3941: Pinot server side change to
optimize LLC segment completion with direct metadata upload.
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r267025969
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
##
@@ -681,11 +690,23 @@ protected SegmentBuildDescriptor
buildSegmentInternal(boolean forCommit) {
TimeUnit.MILLISECONDS.toSeconds(waitTimeMillis));
if (forCommit) {
+Preconditions.checkState(destDir.listFiles() != null &&
destDir.listFiles().length > 0,
+"The index dir is empty: ", destDir);
+File metadataFileName = new File(destDir.listFiles()[0],
V1Constants.MetadataKeys.METADATA_FILE_NAME);
Review comment:
destDir.listFiles() is being called 4 times. Might be cleaner if we store it
in a local variable
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.
npawar commented on a change in pull request #3941: Pinot server side change to
optimize LLC segment completion with direct metadata upload.
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r267018656
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
##
@@ -681,11 +690,23 @@ protected SegmentBuildDescriptor
buildSegmentInternal(boolean forCommit) {
TimeUnit.MILLISECONDS.toSeconds(waitTimeMillis));
if (forCommit) {
+Preconditions.checkState(destDir.listFiles() != null &&
destDir.listFiles().length > 0,
Review comment:
Similar to other exceptions in segment build path, let's catch these
exceptions and return null.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.
npawar commented on a change in pull request #3941: Pinot server side change to
optimize LLC segment completion with direct metadata upload.
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r267024575
##
File path:
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManagerConfig.java
##
@@ -163,6 +165,11 @@ public boolean isEnableSplitCommit() {
return _instanceDataManagerConfiguration.getBoolean(ENABLE_SPLIT_COMMIT,
false);
}
+ @Override
+ public boolean isEnableSplitCommitEndWithMetadata() {
+return
_instanceDataManagerConfiguration.getBoolean(ENABLE_SPLIT_COMMIT_END_WITH_METADATA,
true);
+ }
Review comment:
shouldn't this be default false?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli opened a new pull request #3986: Modify documentation in managing Pinot page
jackjlli opened a new pull request #3986: Modify documentation in managing Pinot page URL: https://github.com/apache/incubator-pinot/pull/3986 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch add-docs updated (1cb4fe5 -> 9b50d89)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch add-docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 1cb4fe5 Add documentation new 9b50d89 Add documentation This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (1cb4fe5) \ N -- N -- N refs/heads/add-docs (9b50d89) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 5845 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/getting_started.rst | 7 --- 1 file changed, 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh merged pull request #3985: [TE] frontend - harleyjj/report-anomaly - adds back report-anomaly mo…
jihaozh merged pull request #3985: [TE] frontend - harleyjj/report-anomaly - adds back report-anomaly mo… URL: https://github.com/apache/incubator-pinot/pull/3985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [TE] frontend - harleyjj/preview - default preview to 2 days to accomodate daily metrics (#3980)
This is an automated email from the ASF dual-hosted git repository.
jihao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new df62374 [TE] frontend - harleyjj/preview - default preview to 2 days
to accomodate daily metrics (#3980)
df62374 is described below
commit df6237481d76952d08b6307be2d6ba2510e0a679
Author: Harley Jackson
AuthorDate: Tue Mar 19 10:22:55 2019 -0700
[TE] frontend - harleyjj/preview - default preview to 2 days to accomodate
daily metrics (#3980)
---
.../thirdeye-frontend/app/pods/components/alert-details/component.js| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
b/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
index aa6be01..b70b5d5 100644
--- a/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
+++ b/thirdeye/thirdeye-frontend/app/pods/components/alert-details/component.js
@@ -48,7 +48,7 @@ export default Component.extend({
anomalyMapping: {},
timeseries: null,
isLoading: false,
- analysisRange: [moment().subtract(1, 'day').startOf('day').valueOf(),
moment().startOf('day').valueOf()],
+ analysisRange: [moment().subtract(1, 'day').startOf('day').valueOf(),
moment().add(1, 'day').startOf('day').valueOf()],
isPendingData: false,
colorMapping: colorMapping,
zoom: {
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh merged pull request #3980: [TE] frontend - harleyjj/preview - default preview to 2 days to accom…
jihaozh merged pull request #3980: [TE] frontend - harleyjj/preview - default preview to 2 days to accom… URL: https://github.com/apache/incubator-pinot/pull/3980 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch nested-object-indexing updated: fixing license header
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch nested-object-indexing in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/nested-object-indexing by this push: new c2d822d fixing license header c2d822d is described below commit c2d822d8fd16d0e57d6b0f36c557ea89617f1a60 Author: Xiang Fu AuthorDate: Mon Mar 18 23:49:59 2019 -0700 fixing license header --- .../pinot/integration/tests/LuceneRealtimeTest.java| 18 ++ 1 file changed, 18 insertions(+) diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/LuceneRealtimeTest.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/LuceneRealtimeTest.java index e2418b6..46e53fa 100644 --- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/LuceneRealtimeTest.java +++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/LuceneRealtimeTest.java @@ -1,3 +1,21 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ package org.apache.pinot.integration.tests; import java.io.File; - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch nested-object-indexing updated (14b9397 -> 635c451)
This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a change to branch nested-object-indexing in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 14b9397 Fixing license header discard be67929 Adding support for bytes type in realtime + nested object indexing discard fa3ba9b Wiring up end to end to support indexing nested fields on complex objects discard 6ad8a96 Adding support for Object Type discard a055202 Enhancing PQL to support MATCHES predicate, can be used for searching within text, map, json and other complex objects discard e89e637 Adding support for MATCHES Predicate new 6adfa98 When counting running segments for tasks, only consider tasks scheduled in one day (#3848) new 0675aee [TE] Fix bugs in user dashboard endpoint (#3853) new 61bfdc6 [TE] Endpoint for migrating partially migrated alerts (#3855) new 0d981ab [TE] detection - time series provider loading cache (#3842) new c943843 Minor fix for integration test (#3858) new 50d3eae [TE] Update Yaml templates (#3859) new bfce886 Fix compile failure issue when docs/_build is present (#3857) new a745087 Add header definition for rst files (#3860) new 795f494 [TE] frontend - harleyjj/rootcause-anomaly - never show inconsistent data warning for 1 and 5 minute granularity (#3861) new b2c3528 Add scheduler wait time for server side log (#3862) new 4f96cba [TE] frontend - harleyjj/alert-details - multiple alert overview improvements for MVP (#3863) new 23cda0b only show inconsistent alert when it is daily metrics (#3865) new 1f9d096 [TE] detection - task scheduler backoff fix (#3866) new b3f7fee Fix reporting/logging of numEntriesScannedPostFilter (#3870) new f70935e Disable realtime aggregate metrics if multi value columns present (#3868) new e3a9184 Fix listFiles method in HadoopPinotFS (#3838) new de950ff [TE] Update label and detection description (#3878) new 126b022 [TE] Include alert description in the email (#3879) new e6cdea0 [TE] frontend - harleyjj/manage-alerts - fix display bugs in alerts list (#3882) new 209c966 [TE] detection - threshold-based filter (#3881) new 4e7ffb3 [TE] Migration - fill in metric urn for old anomalies (#3880) new 2da1157 Doc coding guidelines (#3884) new 47be0f0 [TE] Enable description after migration (#3883) new 9269f5e Check assigned instances for uploading new segment (#3856) new 1f2c2d7 Move images files to img directory for docs (#3886) new 8f647cf Unify move method in PinotFS (#3834) new 52a9559 [TE] frontend - fix some bugs with alert details graph on range change (#3876) new 3b09d27 [TE] Endpoint to delete a complete application (#3888) new 3b92bee [TE] Minor fix in template (#3889) new 6fc48ce [TE] frontend - harleyjj/alert-details - limit number of anomalies displayed in graph (#3890) new caef8f6 Replace deprecated ZkClient APIs (#3894) new 9791a3d [TE] frondend - hide anomaly legend if the anomalies is too much. (#3896) new 4964f91 Actively check cluster changes if there is no callback for a long time (#3885) new 97f0a86 hadern DateTimeFormatSpecTest with locale (#3785) new 9dda23f In ClusterChangeMediator, remove sleep and make it notify based (#3898) new 4dc59ff Reduce MAX_SEGMENT_COMPLETION_TIME_MILLIS to 5 minutes (#3893) new ca72055 [TE] migration - bucket period conversion (#3902) new 8bf4011 [TE] Hide Linkedin URLs and add an inlab wiki reference for creating an alert in UI (#3901) new ce551eb [TE] Add cron validator and make detection cron configurable (#3903) new bab7926 [TE] frontend - explore dimensions in new alert overview (#3897) new 3b95a64 [TE] frontend harleyjj/create-alert turn form off for create-alert (#3905) new ec22501 Update read-the-docs high level organization. More changes to follow (#3906) new 0e111e6 Remove references that we don't need (#3907) new 91237aa [TE] pinot -harleyjj/email-content update view/edit linke to edit-new route (#3909) new dbf02c8 [TE] frontend - harleyjj/alert-overview - handle empty responses better (#3908) new a7a8bb6 Fix the RecordReader to pick incoming time column (#3895) new 0c59c28 [TE] frontend - harleyjj/alert-overview - fix response parsing error in alert overview and edit alert (#3911) new 867edcf Update the display for sidebar navigation to include section names (#3910) new 06da717 Log actual configurations for SegmentCreationMapper (#3914) new cda36d7 [TE] Option to retain/force delete anomalies during replay (#3916) new 6a1feab [TE] detection - merger max duration (#3915) new db89922 [TE] detection - add default values for duration filter (#3917) new b3bd35b Fix SegmentGeneratorConfig where the time format in schema is not picked up
