[incubator-pinot] branch add-docs updated (5176b82 -> e215a6f)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch add-docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


 discard 5176b82  Add documentation
 new e215a6f  Add documentation

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (5176b82)
\
 N -- N -- N   refs/heads/add-docs (e215a6f)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 5845 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 docs/img/list-schemas.png | Bin 8952 -> 247946 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch nested-object-indexing-1 created (now bdaa9c1)

[xiangfu]

This is an automated email from the ASF dual-hosted git repository.

xiangfu pushed a change to branch nested-object-indexing-1
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


  at bdaa9c1  Fixing license header

This branch includes the following new commits:

 new a1d2d69  Adding support for MATCHES Predicate
 new 91257aa  Enhancing PQL to support MATCHES predicate, can be used for 
searching within text, map, json and other complex objects
 new 74f8e6e  Adding support for Object Type
 new 224299d  Wiring up end to end to support indexing nested fields on 
complex objects
 new a230c64  Adding support for bytes type in realtime + nested object 
indexing
 new bdaa9c1  Fixing license header

The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] 05/06: Adding support for bytes type in realtime + nested object indexing

[xiangfu]

This is an automated email from the ASF dual-hosted git repository.

xiangfu pushed a commit to branch nested-object-indexing-1
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git

commit a230c647551bf5de33646f58edeb690be2e54186
Author: kishore gopalakrishna 
AuthorDate: Tue Feb 19 09:29:40 2019 -0800

Adding support for bytes type in realtime + nested object indexing
---
 .../org/apache/pinot/common/data/FieldSpec.java|   1 +
 .../org/apache/pinot/common/data/PinotObject.java  |   2 +-
 .../pinot/common/data/PinotObjectFactory.java  |  67 +
 .../org/apache/pinot/core/common/Predicate.java|  28 +---
 .../realtime/LLRealtimeSegmentDataManager.java |   3 +-
 .../indexsegment/mutable/MutableSegmentImpl.java   | 149 -
 ...VarByteSingleColumnSingleValueReaderWriter.java | 143 
 .../core/realtime/impl/RealtimeSegmentConfig.java  |  18 ++-
 .../invertedindex/RealtimeInvertedIndexReader.java |  41 +-
 .../creator/impl/inv/LuceneIndexCreator.java   |  14 +-
 .../loader/invertedindex/InvertedIndexHandler.java |  29 +---
 .../index/readers/LuceneInvertedIndexReader.java   |   9 +-
 ...yteSingleColumnSingleValueReaderWriterTest.java |  31 +
 13 files changed, 433 insertions(+), 102 deletions(-)

diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java 
b/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java
index ad5a5a3..cb43a4c 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java
@@ -457,6 +457,7 @@ public abstract class FieldSpec implements 
Comparable, ConfigNodeLife
 case DOUBLE:
   return Double.BYTES;
 case BYTES:
+case STRING:
   // TODO: Metric size is only used for Star-tree generation, which is 
not supported yet.
   return MetricFieldSpec.UNDEFINED_METRIC_SIZE;
 default:
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObject.java 
b/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObject.java
index 3f1ca33..9b68f73 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObject.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObject.java
@@ -50,7 +50,7 @@ public interface PinotObject {
   List getPropertyNames();
 
   /**
-   * @param fieldName
+   * @param propertyName
* @return the value of the property, it can be a single object or a list of 
objects.
*/
   Object getProperty(String propertyName);
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObjectFactory.java
 
b/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObjectFactory.java
new file mode 100644
index 000..b15dc58
--- /dev/null
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObjectFactory.java
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.data;
+
+import java.util.List;
+import org.apache.pinot.common.data.objects.JSONObject;
+import org.apache.pinot.common.data.objects.MapObject;
+import org.apache.pinot.common.data.objects.TextObject;
+import org.apache.pinot.common.segment.fetcher.HdfsSegmentFetcher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Factory class that create PinotObject from bytes
+ */
+public class PinotObjectFactory {
+
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(PinotObjectFactory.class);
+
+  public static PinotObject create(FieldSpec spec, byte[] buf) {
+return create(spec.getObjectType(), buf);
+  }
+
+  public static PinotObject create(String objectType, byte[] buf) {
+
+Class pinotObjectClazz;
+PinotObject pinotObject = null;
+try {
+  switch (objectType.toUpperCase()) {
+case "MAP":
+  pinotObjectClazz = MapObject.class;
+  break;
+case "JSON":
+  pinotObjectClazz = JSONObject.class;
+  break;
+case "TEXT":
+  pinotObjectClazz = TextObject.class;
+  break;
+default:
+  // custom object type.
+  

[incubator-pinot] 03/06: Adding support for Object Type

[xiangfu]

This is an automated email from the ASF dual-hosted git repository.

xiangfu pushed a commit to branch nested-object-indexing-1
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git

commit 74f8e6eede16c3a8eba97d449feeeb70d486b76b
Author: kishore gopalakrishna 
AuthorDate: Sun Jan 20 17:49:05 2019 -0800

Adding support for Object Type
---
 .../org/apache/pinot/common/data/FieldSpec.java| 23 +-
 .../org/apache/pinot/common/data/PinotObject.java  | 58 +++
 .../pinot/common/data/objects/JSONObject.java  | 83 ++
 .../pinot/common/data/objects/MapObject.java   | 66 +
 .../pinot/common/data/objects/TextObject.java  | 53 ++
 .../creator/impl/SegmentColumnarIndexCreator.java  |  1 +
 .../core/segment/creator/impl/V1Constants.java |  1 +
 .../pinot/core/segment/index/ColumnMetadata.java   | 21 --
 8 files changed, 301 insertions(+), 5 deletions(-)

diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java 
b/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java
index 30be748..080f0e7 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java
@@ -86,6 +86,10 @@ public abstract class FieldSpec implements 
Comparable, ConfigNodeLife
 
   @ConfigKey("virtualColumnProvider")
   protected String _virtualColumnProvider;
+  
+  //Complex type that can be constructed from raw bytes stored e.g. map, json, 
text
+  @ConfigKey("objectType")
+  protected String _objectType;
 
   // Default constructor required by JSON de-serializer. DO NOT REMOVE.
   public FieldSpec() {
@@ -98,15 +102,21 @@ public abstract class FieldSpec implements 
Comparable, ConfigNodeLife
   public FieldSpec(String name, DataType dataType, boolean isSingleValueField, 
@Nullable Object defaultNullValue) {
 this(name, dataType, isSingleValueField, DEFAULT_MAX_LENGTH, 
defaultNullValue);
   }
-
+  
   public FieldSpec(String name, DataType dataType, boolean isSingleValueField, 
int maxLength,
   @Nullable Object defaultNullValue) {
+this(name, dataType, isSingleValueField, maxLength, defaultNullValue, 
null);
+  }
+  
+  public FieldSpec(String name, DataType dataType, boolean isSingleValueField, 
int maxLength,
+  @Nullable Object defaultNullValue, @Nullable String objectType) {
 _name = name;
 _dataType = dataType.getStoredType();
 _isSingleValueField = isSingleValueField;
 _maxLength = maxLength;
 setDefaultNullValue(defaultNullValue);
   }
+  
 
   public abstract FieldType getFieldType();
 
@@ -183,6 +193,16 @@ public abstract class FieldSpec implements 
Comparable, ConfigNodeLife
   _defaultNullValue = getDefaultNullValue(getFieldType(), _dataType, 
_stringDefaultNullValue);
 }
   }
+  
+  
+  
+  public String getObjectType() {
+return _objectType;
+  }
+
+  public void setObjectType(String objectType) {
+_objectType = objectType;
+  }
 
   private static Object getDefaultNullValue(FieldType fieldType, DataType 
dataType,
   @Nullable String stringDefaultNullValue) {
@@ -353,6 +373,7 @@ public abstract class FieldSpec implements 
Comparable, ConfigNodeLife
 result = EqualityUtils.hashCodeOf(result, _isSingleValueField);
 result = EqualityUtils.hashCodeOf(result, 
getStringValue(_defaultNullValue));
 result = EqualityUtils.hashCodeOf(result, _maxLength);
+result = EqualityUtils.hashCodeOf(result, _objectType);
 return result;
   }
 
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObject.java 
b/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObject.java
new file mode 100644
index 000..3f1ca33
--- /dev/null
+++ b/pinot-common/src/main/java/org/apache/pinot/common/data/PinotObject.java
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.data;
+
+import java.util.List;
+
+/**
+ * Common interface for complex Object types such as HyperLogLog, Map, JSON 
etc.
+ * Flow to convert byte[] to PinotObject
+ * - compute the objectTypeClass from objectType (from 

[incubator-pinot] 01/06: Adding support for MATCHES Predicate

[xiangfu]

This is an automated email from the ASF dual-hosted git repository.

xiangfu pushed a commit to branch nested-object-indexing-1
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git

commit a1d2d690221686a8073dfbadd015d5eecea0a6a4
Author: kishore gopalakrishna 
AuthorDate: Fri Jan 18 16:05:45 2019 -0800

Adding support for MATCHES Predicate
---
 .../antlr4/org/apache/pinot/pql/parsers/PQL2.g4|   5 +
 .../apache/pinot/common/config/Deserializer.java   |   5 +-
 .../pinot/common/request/AggregationInfo.java  |  73 +--
 .../apache/pinot/common/request/BrokerRequest.java | 131 +--
 .../pinot/common/request/FilterOperator.java   |  32 ++---
 .../apache/pinot/common/request/FilterQuery.java   |  87 ++---
 .../pinot/common/request/FilterQueryMap.java   |  64 +
 .../org/apache/pinot/common/request/GroupBy.java   | 137 +--
 .../pinot/common/request/HavingFilterQuery.java|  87 ++---
 .../pinot/common/request/HavingFilterQueryMap.java |  62 -
 .../pinot/common/request/InstanceRequest.java  | 111 
 .../apache/pinot/common/request/QuerySource.java   |  60 -
 .../org/apache/pinot/common/request/QueryType.java |  77 +--
 .../org/apache/pinot/common/request/Selection.java | 145 ++---
 .../apache/pinot/common/request/SelectionSort.java |  65 +
 .../pinot/common/response/ProcessingException.java |  66 +-
 .../apache/pinot/pql/parsers/Pql2AstListener.java  |  11 ++
 .../parsers/pql2/ast/MatchesPredicateAstNode.java  |  62 +
 .../org/apache/pinot/core/common/Predicate.java|  35 -
 .../core/common/predicate/MatchesPredicate.java|  50 +++
 .../MatchesPredicateEvaluatorFactory.java  |  69 ++
 .../predicate/PredicateEvaluatorProvider.java  |   6 +-
 22 files changed, 787 insertions(+), 653 deletions(-)

diff --git a/pinot-common/src/main/antlr4/org/apache/pinot/pql/parsers/PQL2.g4 
b/pinot-common/src/main/antlr4/org/apache/pinot/pql/parsers/PQL2.g4
index 5086182..17652ca 100644
--- a/pinot-common/src/main/antlr4/org/apache/pinot/pql/parsers/PQL2.g4
+++ b/pinot-common/src/main/antlr4/org/apache/pinot/pql/parsers/PQL2.g4
@@ -77,6 +77,7 @@ predicate:
   | betweenClause # BetweenPredicate
   | isClause  # IsPredicate
   | regexpLikeClause  # RegexpLikePredicate
+  | matchesClause # MatchesPredicate  
   ;
 
 inClause:
@@ -95,6 +96,9 @@ betweenClause:
 regexpLikeClause:
   REGEXP_LIKE '(' expression ',' literal ')';
 
+matchesClause:
+  expression MATCHES '(' literal ',' literal ')';
+
 booleanOperator: OR | AND;
 
 groupByClause: GROUP BY groupByList;
@@ -131,6 +135,7 @@ LIMIT: L I M I T;
 NOT : N O T;
 OR: O R;
 REGEXP_LIKE: R E G E X P '_' L I K E;
+MATCHES: M A T C H E S;
 ORDER: O R D E R;
 SELECT: S E L E C T;
 TOP: T O P;
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/config/Deserializer.java 
b/pinot-common/src/main/java/org/apache/pinot/common/config/Deserializer.java
index 93c4d84..63646c4 100644
--- 
a/pinot-common/src/main/java/org/apache/pinot/common/config/Deserializer.java
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/config/Deserializer.java
@@ -404,8 +404,9 @@ public class Deserializer {
 config = config.resolve();
 
 try {
-  return deserialize(clazz, 
io.vavr.collection.HashSet.ofAll(config.entrySet())
-  .toMap(entry -> Tuple.of(entry.getKey(), 
entry.getValue().unwrapped())), "");
+  Map map = 
io.vavr.collection.HashSet.ofAll(config.entrySet())
+  .toMap(entry -> Tuple.of(entry.getKey(), 
entry.getValue().unwrapped()));
+  return deserialize(clazz, map, "");
 } catch (Exception e) {
   Utils.rethrowException(e);
   return null;
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/request/AggregationInfo.java
 
b/pinot-common/src/main/java/org/apache/pinot/common/request/AggregationInfo.java
index aa0b3db..ab129ff 100644
--- 
a/pinot-common/src/main/java/org/apache/pinot/common/request/AggregationInfo.java
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/request/AggregationInfo.java
@@ -1,22 +1,4 @@
 /**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *   http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied.  

[incubator-pinot] 04/06: Wiring up end to end to support indexing nested fields on complex objects

[xiangfu]

This is an automated email from the ASF dual-hosted git repository.

xiangfu pushed a commit to branch nested-object-indexing-1
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git

commit 224299deccf308ddfdab2e59556d413944596329
Author: kishore gopalakrishna 
AuthorDate: Thu Jan 24 19:53:29 2019 -0800

Wiring up end to end to support indexing nested fields on complex objects
---
 .../org/apache/pinot/common/data/FieldSpec.java|   3 +
 .../pinot/common/data/objects/TextObject.java  |   3 +-
 pinot-core/pom.xml |  19 ++-
 .../immutable/ImmutableSegmentLoader.java  |   4 +-
 .../io/reader/impl/v1/SortedIndexReaderImpl.java   |   6 +
 .../core/operator/filter/FilterOperatorUtils.java  |   8 +-
 .../filter/IndexBasedMatchesFilterOperator.java|  86 +++
 .../filter/ScanBasedMatchesFilterOperator.java |  33 ++--
 .../MatchesPredicateEvaluatorFactory.java  |  21 +--
 .../invertedindex/RealtimeInvertedIndexReader.java |   8 +-
 .../core/segment/creator/InvertedIndexCreator.java |   8 +
 .../core/segment/creator/impl/V1Constants.java |   1 +
 .../creator/impl/inv/LuceneIndexCreator.java   | 127 +++
 .../inv/OffHeapBitmapInvertedIndexCreator.java |   6 +
 .../impl/inv/OnHeapBitmapInvertedIndexCreator.java |   6 +
 .../index/column/PhysicalColumnIndexContainer.java |  22 ++-
 .../index/data/source/ColumnDataSource.java|   2 +-
 .../loader/invertedindex/InvertedIndexHandler.java | 106 -
 .../index/readers/BitmapInvertedIndexReader.java   |   7 +
 .../segment/index/readers/InvertedIndexReader.java |   8 +
 .../index/readers/LuceneInvertedIndexReader.java   | 157 +++
 .../virtualcolumn/DocIdVirtualColumnProvider.java  |   7 +-
 .../SingleStringVirtualColumnProvider.java |   7 +-
 .../tests/LuceneIndexClusterIntegrationTest.java   | 172 +
 pinot-perf/pom.xml |  35 -
 .../org/apache/pinot/perf/LuceneBenchmark.java |  80 ++
 .../apache/pinot/tools/perf/ZookeeperLauncher.java |   2 +-
 27 files changed, 887 insertions(+), 57 deletions(-)

diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java 
b/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java
index 080f0e7..ad5a5a3 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/data/FieldSpec.java
@@ -335,6 +335,9 @@ public abstract class FieldSpec implements 
Comparable, ConfigNodeLife
   case STRING:
 jsonSchema.set("type", convertStringsToJsonArray("null", "string"));
 return jsonSchema;
+  case BYTES:
+jsonSchema.set("type", convertStringsToJsonArray("null", "bytes"));
+return jsonSchema;
   default:
 throw new UnsupportedOperationException();
 }
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/data/objects/TextObject.java
 
b/pinot-common/src/main/java/org/apache/pinot/common/data/objects/TextObject.java
index cf5f27e..c171c40 100644
--- 
a/pinot-common/src/main/java/org/apache/pinot/common/data/objects/TextObject.java
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/data/objects/TextObject.java
@@ -27,7 +27,8 @@ import com.google.common.collect.Lists;
 public class TextObject implements PinotObject {
 
   byte[] _bytes;
-  private static List _FIELDS = Lists.newArrayList("Content");
+  public static String DEFAULT_FIELD = "Content";
+  private static List _FIELDS = Lists.newArrayList(DEFAULT_FIELD);
 
   @Override
   public void init(byte[] bytes) {
diff --git a/pinot-core/pom.xml b/pinot-core/pom.xml
index 8f4efe2..4c8215a 100644
--- a/pinot-core/pom.xml
+++ b/pinot-core/pom.xml
@@ -196,7 +196,24 @@
 
   
 
-
+
+
+
+  org.apache.lucene
+  lucene-core
+  7.6.0
+
+
+  org.apache.lucene
+  lucene-queryparser
+  7.6.0
+
+
+  org.apache.lucene
+  lucene-analyzers-common
+  7.6.0
+
+
 
 
   org.mockito
diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegmentLoader.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegmentLoader.java
index 2ddb80c..3d7ac31 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegmentLoader.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/immutable/ImmutableSegmentLoader.java
@@ -118,8 +118,8 @@ public class ImmutableSegmentLoader {
 SegmentDirectory.Reader segmentReader = segmentDirectory.createReader();
 Map indexContainerMap = new HashMap<>();
 for (Map.Entry entry : 
segmentMetadata.getColumnMetadataMap().entrySet()) {
-  indexContainerMap
-  .put(entry.getKey(), new PhysicalColumnIndexContainer(segmentReader, 
entry.getValue(), 

[incubator-pinot] 06/06: Fixing license header

[xiangfu]

This is an automated email from the ASF dual-hosted git repository.

xiangfu pushed a commit to branch nested-object-indexing-1
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git

commit bdaa9c12938254819c09c05b7a589a24fd8c860d
Author: Xiang Fu 
AuthorDate: Mon Mar 18 22:27:55 2019 -0700

Fixing license header
---
 ...VarByteSingleColumnSingleValueReaderWriterTest.java | 18 ++
 1 file changed, 18 insertions(+)

diff --git 
a/pinot-core/src/test/java/org/apache/pinot/core/io/writer/impl/VarByteSingleColumnSingleValueReaderWriterTest.java
 
b/pinot-core/src/test/java/org/apache/pinot/core/io/writer/impl/VarByteSingleColumnSingleValueReaderWriterTest.java
index 0fc33db..2f1f386 100644
--- 
a/pinot-core/src/test/java/org/apache/pinot/core/io/writer/impl/VarByteSingleColumnSingleValueReaderWriterTest.java
+++ 
b/pinot-core/src/test/java/org/apache/pinot/core/io/writer/impl/VarByteSingleColumnSingleValueReaderWriterTest.java
@@ -1,3 +1,21 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
 package org.apache.pinot.core.io.writer.impl;
 
 import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch nested-object-indexing updated: Fixing license header

[xiangfu]

This is an automated email from the ASF dual-hosted git repository.

xiangfu pushed a commit to branch nested-object-indexing
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/nested-object-indexing by this 
push:
 new 14b9397  Fixing license header
14b9397 is described below

commit 14b9397c06e4a56bca503d5902a7fffa087dfb56
Author: Xiang Fu 
AuthorDate: Mon Mar 18 22:27:55 2019 -0700

Fixing license header
---
 ...VarByteSingleColumnSingleValueReaderWriterTest.java | 18 ++
 1 file changed, 18 insertions(+)

diff --git 
a/pinot-core/src/test/java/org/apache/pinot/core/io/writer/impl/VarByteSingleColumnSingleValueReaderWriterTest.java
 
b/pinot-core/src/test/java/org/apache/pinot/core/io/writer/impl/VarByteSingleColumnSingleValueReaderWriterTest.java
index 0fc33db..2f1f386 100644
--- 
a/pinot-core/src/test/java/org/apache/pinot/core/io/writer/impl/VarByteSingleColumnSingleValueReaderWriterTest.java
+++ 
b/pinot-core/src/test/java/org/apache/pinot/core/io/writer/impl/VarByteSingleColumnSingleValueReaderWriterTest.java
@@ -1,3 +1,21 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
 package org.apache.pinot.core.io.writer.impl;
 
 import org.apache.pinot.core.io.readerwriter.PinotDataBufferMemoryManager;


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] harleyjj opened a new pull request #3985: [TE] frontend - harleyjj/report-anomaly - adds back report-anomaly mo…

[GitBox]

harleyjj opened a new pull request #3985: [TE] frontend - 
harleyjj/report-anomaly - adds back report-anomaly mo…
URL: https://github.com/apache/incubator-pinot/pull/3985
 
 
   …dal to alert overview
   
   1) Updates report-anomaly modal for new pipeline and adds it to new alert 
overview
   2) Updates feedback and anomaly urls to new endpoints
   3) Cleans up some eslint warnings
   4) Fixes bug with edit alert page


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266700535
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,32 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
+
+More info on metrics collection and viewing a complete set of available metric 
is available in the `Metrics `_ section.
+
+Pinot Server
+
+* Missing Segments - Number of missing segments - `NUM_MISSING_SEGMENTS 
`_
 
 Review comment:
   Probably good to give units as well where applicable (for example for 
consumption status, boolean might be confusing)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266704832
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently 
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and 
``replicasPerPartition`` for realtime tables will indicate how many replicas of 
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "ugcGestureEvents",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of 
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list 
of real column names as specified in the schema. Column names present will NOT 
have a dictionary created. More info on indexes can be found on the `Index 
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column 
names as specified in the schema.
+* Aggregate Metrics - Using ``aggregateMetrics`` set to ``"true"`` to enable 
the feature and ``"false"`` to disable. This feature is only available on 
REALTIME tables.
+* Data Partitioning Strategy using the ``segmentPartitionConfig`` to configure 

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266701093
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,32 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
+
+More info on metrics collection and viewing a complete set of available metric 
is available in the `Metrics `_ section.
+
+Pinot Server
+
+* Missing Segments - Number of missing segments - `NUM_MISSING_SEGMENTS 
`_
+* Query latency - Latency from the time a server receives a request to when it 
sends a response - `TOTAL_QUERY_TIME 
`_
+* Query Execution Exceptions - The number of exception which might have 
occurred during query execution - `QUERY_EXECUTION_EXCEPTIONS 
`_
+* Realtime Consumption Status - It's important to ensure at least a single 
replica of each partition is consuming - `LLC_PARTITION_CONSUMING 
`_
+* Realtime Highest Offset Consumed - `HIGHEST_STREAM_OFFSET_CONSUMED 
`_
+
+Pinot Broker
+
+* Incoming QPS (per broker) - `QUERIES 
`_
+* Dropped Requests - `REQUEST_DROPPED_DUE_TO_SEND_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_CONNECTION_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_ACCESS_ERROR 
`_
+* Partial Responses - `BROKER_RESPONSES_WITH_PARTIAL_SERVERS_RESPONDED 
`_
+* Table QPS quota exceeded - `QUERY_QUOTA_EXCEEDED 
`_
+* Table QPS quota usage percent - `QUERY_QUOTA_CAPACITY_UTILIZATION_RATE 
`_
+
+Pinot Controller
+
+* Missing Segment Count -
 
 Review comment:
   Gaps in time for segment coverage, I think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266703680
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently 
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and 
``replicasPerPartition`` for realtime tables will indicate how many replicas of 
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "ugcGestureEvents",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of 
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list 
of real column names as specified in the schema. Column names present will NOT 
have a dictionary created. More info on indexes can be found on the `Index 
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column 
names as specified in the schema.
 
 Review comment:
   Does this only affect realtime?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go 

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266701213
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,32 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
+
+More info on metrics collection and viewing a complete set of available metric 
is available in the `Metrics `_ section.
+
+Pinot Server
+
+* Missing Segments - Number of missing segments - `NUM_MISSING_SEGMENTS 
`_
+* Query latency - Latency from the time a server receives a request to when it 
sends a response - `TOTAL_QUERY_TIME 
`_
+* Query Execution Exceptions - The number of exception which might have 
occurred during query execution - `QUERY_EXECUTION_EXCEPTIONS 
`_
+* Realtime Consumption Status - It's important to ensure at least a single 
replica of each partition is consuming - `LLC_PARTITION_CONSUMING 
`_
+* Realtime Highest Offset Consumed - `HIGHEST_STREAM_OFFSET_CONSUMED 
`_
+
+Pinot Broker
+
+* Incoming QPS (per broker) - `QUERIES 
`_
+* Dropped Requests - `REQUEST_DROPPED_DUE_TO_SEND_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_CONNECTION_ERROR 
`_,
 `REQUEST_DROPPED_DUE_TO_ACCESS_ERROR 
`_
+* Partial Responses - `BROKER_RESPONSES_WITH_PARTIAL_SERVERS_RESPONDED 
`_
+* Table QPS quota exceeded - `QUERY_QUOTA_EXCEEDED 
`_
+* Table QPS quota usage percent - `QUERY_QUOTA_CAPACITY_UTILIZATION_RATE 
`_
+
+Pinot Controller
+
+* Missing Segment Count -
+* Segments in Error State -
 
 Review comment:
   Hopefully we have some information somewhere about what ERROR state means? 
or how to find what caused the ERROR state (ie instance errors) -- it would be 
good to link to that here


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266703153
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
 
 Review comment:
   * Allowed values or examples as applicable


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266703992
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently 
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and 
``replicasPerPartition`` for realtime tables will indicate how many replicas of 
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "ugcGestureEvents",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of 
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list 
of real column names as specified in the schema. Column names present will NOT 
have a dictionary created. More info on indexes can be found on the `Index 
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column 
names as specified in the schema.
+* Aggregate Metrics - Using ``aggregateMetrics`` set to ``"true"`` to enable 
the feature and ``"false"`` to disable. This feature is only available on 
REALTIME tables.
 
 Review comment:
   Maybe there should eventually be a page for this feature

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266703613
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently 
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and 
``replicasPerPartition`` for realtime tables will indicate how many replicas of 
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "ugcGestureEvents",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of 
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list 
of real column names as specified in the schema. Column names present will NOT 
have a dictionary created. More info on indexes can be found on the `Index 
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column 
names as specified in the schema.
+* Aggregate Metrics - Using ``aggregateMetrics`` set to ``"true"`` to enable 
the feature and ``"false"`` to disable. This feature is only available on 
REALTIME tables.
 
 Review comment:
   What does this do? Just a brief overview, unless there's a page 

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266703950
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently 
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and 
``replicasPerPartition`` for realtime tables will indicate how many replicas of 
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "ugcGestureEvents",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of 
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list 
of real column names as specified in the schema. Column names present will NOT 
have a dictionary created. More info on indexes can be found on the `Index 
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column 
names as specified in the schema.
+* Aggregate Metrics - Using ``aggregateMetrics`` set to ``"true"`` to enable 
the feature and ``"false"`` to disable. This feature is only available on 
REALTIME tables.
 
 Review comment:
   You also need all metrics(?) columns in no dictionary columns 

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266705683
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
 
 Review comment:
   Consider rewording.
   Some of this seems unnecessary? There should probably be a section long 
before this discussing the data model of offline and realtime tables.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266703532
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently 
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and 
``replicasPerPartition`` for realtime tables will indicate how many replicas of 
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "ugcGestureEvents",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of 
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list 
of real column names as specified in the schema. Column names present will NOT 
have a dictionary created. More info on indexes can be found on the `Index 
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column 
names as specified in the schema.
+* Aggregate Metrics - Using ``aggregateMetrics`` set to ``"true"`` to enable 
the feature and ``"false"`` to disable. This feature is only available on 
REALTIME tables.
+* Data Partitioning Strategy using the ``segmentPartitionConfig`` to configure 

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266704733
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
 
 Review comment:
   Consider including recommended values where it makes sense? Could go either 
way on this though...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266704620
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently 
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and 
``replicasPerPartition`` for realtime tables will indicate how many replicas of 
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "ugcGestureEvents",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of 
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list 
of real column names as specified in the schema. Column names present will NOT 
have a dictionary created. More info on indexes can be found on the `Index 
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column 
names as specified in the schema.
+* Aggregate Metrics - Using ``aggregateMetrics`` set to ``"true"`` to enable 
the feature and ``"false"`` to disable. This feature is only available on 
REALTIME tables.
+* Data Partitioning Strategy using the ``segmentPartitionConfig`` to configure 

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266704156
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
+
+* Segment Retention - with the  ``retentionTimeUnit`` and 
``retentionTimeValue`` options.
+* Segment Push - Using ``segmentPushFrequency`` to indicate how frequently 
segments are uploaded.
+* Replication - Using ``replication`` for offline tables and 
``replicasPerPartition`` for realtime tables will indicate how many replicas of 
data will be present.
+* Schema - The name of the schema that's been uploaded to the controller
+* Time column - using ``timeColumnName`` and ``timeType``, this must match 
what's configured in the preceeding schema
+* Segment assignment strategy - Described more on the page `Customizing Pinot 
`_
+
+
+.. code-block:: none
+
+"segmentsConfig": {
+  "retentionTimeUnit": "DAYS",
+  "retentionTimeValue": "5",
+  "segmentPushFrequency": "daily",
+  "segmentPushType": "APPEND",
+  "replication": "3",
+  "replicasPerPartition": "3",
+  "schemaName": "ugcGestureEvents",
+  "timeColumnName": "daysSinceEpoch",
+  "timeType": "DAYS",
+  "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
+},
+
+Table Index Config Section
+~~
+
+The ``tableIndexConfig`` section has information about how to configure:
+
+* Inverted Indexes - Using the ``invertedIndexColumns`` to specify a list of 
real column names as specified in the schema.
+* No Dictionary Columns - Using the ``noDictionaryColumns`` to specify a list 
of real column names as specified in the schema. Column names present will NOT 
have a dictionary created. More info on indexes can be found on the `Index 
Techniques `_ page.
+* Sorted Column - Using the ``sortedColumn`` to specify a list of real column 
names as specified in the schema.
+* Aggregate Metrics - Using ``aggregateMetrics`` set to ``"true"`` to enable 
the feature and ``"false"`` to disable. This feature is only available on 
REALTIME tables.
+* Data Partitioning Strategy using the ``segmentPartitionConfig`` to configure 

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266701869
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -40,11 +40,14 @@ Recommended environment
 Deploying Pinot
 ---
 
-Direct deployment of Pinot
-~~
+In general, when deploying Pinot services, it is best to adhere to a specific 
ordering in which the various components should be deployed. This deployment 
order is recommended incase of the scenario that there might be protocol or 
other significant differences, the deployments go out in a predictable order in 
which failure  due to these changes can be avoided.
 
 Review comment:
   Consider wrapping long lines at 79 characters.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266699836
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,32 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
+
+More info on metrics collection and viewing a complete set of available metric 
is available in the `Metrics `_ section.
+
+Pinot Server
+
+* Missing Segments - Number of missing segments - `NUM_MISSING_SEGMENTS 
`_
 
 Review comment:
   This explanation is confusing.
   
   Metric is number of missing segments that the broker queried for (expected 
to be on the server) but the server didn't have. This can be due to retention 
or stale routing table.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266702135
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
 
 Review comment:
   ```suggestion
   Introduction
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266699635
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,32 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
+
+More info on metrics collection and viewing a complete set of available metric 
is available in the `Metrics `_ section.
+
+Pinot Server
 
 Review comment:
   Probably want to use subheadings


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266702485
 
 

 ##
 File path: docs/tableconfig_schema.rst
 ##
 @@ -0,0 +1,172 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+..   http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Table Config
+===
+
+Table Config
+-
+
+Introduction to table configs
+~
+
+Using tables is how Pinot serves and organizes data. There are many settings 
in the table config which will influence how Pinot operates. The first and most 
significant distinction is using an offline versus a realtime table.
+
+An offline table in Pinot is used to host data which might be periodically 
uploaded - daily, weekly, etc. A realtime table, however, is used to consume 
data from incoming data streams and serve this data in a near-realtime manner. 
This might also be referred to as nearline or just plain 'realtime'.
+
+In this section a sample table configuration will be shown and all sections 
will be explained and if applicable have appropriate sections linked to for 
further explanation of those corresponding Pinot features.
+
+Sample table config and descriptions
+
+
+A sample table config is shown below which has sub-sections collasped. The sub 
sections will be described individually in following sections.
+
+The ``tableName`` should only contain alpha-numeric characters, hyphens ('-'), 
or underscores ('_'). Though using a double-underscore ('__') is not allowed 
and reserved for other features within Pinot.
+
+The ``tableType`` will indicate the type of the table, ``OFFLINE`` or 
``REALTIME``. There are some settings specific to each type. This 
differentiation will be called out below as options are explained.
+
+.. code-block:: none
+
+{
+  "tableName": "myPinotTable",
+  "tableType": "REALTIME"
+  "segmentsConfig": {},
+  "tableIndexConfig": {},
+  "tenants": {},
+  "routing": {},
+  "task": {},
+  "metadata": {}
+}
+
+Segments Config Section
+~~~
+
+The ``segmentsConfig`` section has information about configuring
 
 Review comment:
   configuring...?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266701369
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,32 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
+
+More info on metrics collection and viewing a complete set of available metric 
is available in the `Metrics `_ section.
+
+Pinot Server
+
+* Missing Segments - Number of missing segments - `NUM_MISSING_SEGMENTS 
`_
 
 Review comment:
   Also consider ordering metrics by importance.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] dinoocch commented on a change in pull request #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

dinoocch commented on a change in pull request #3975: ReadTheDocs documentation 
for Table Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#discussion_r266700385
 
 

 ##
 File path: docs/in_production.rst
 ##
 @@ -64,4 +67,32 @@ Configuring realtime data ingestion
 Monitoring Pinot
 
 
+In order for Pinot to provide effective service there is a core set of metrics 
which should be monitored to ensure service stability, fault tolerance and 
acceptable response times. In the section following, there are service level 
metrics which are recommended to be monitored.
+
+More info on metrics collection and viewing a complete set of available metric 
is available in the `Metrics `_ section.
+
+Pinot Server
+
+* Missing Segments - Number of missing segments - `NUM_MISSING_SEGMENTS 
`_
 
 Review comment:
   Consider format
   ```
   * : 
   
   * 
   * 
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table 
Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#issuecomment-473083001
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=h1) 
Report
   > Merging 
[#3975](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-pinot/commit/5867b16f32145a80fa98de4d24ec6302b17bb95e?src=pr=desc)
 will **increase** coverage by `0.1%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-pinot/pull/3975/graphs/tree.svg?width=650=4ibza2ugkz=150=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ## master   #3975 +/-   ##
   ==
   + Coverage 66.99%   67.1%   +0.1% 
 Complexity4   4 
   ==
 Files  10321032 
 Lines 51047   51070 +23 
 Branches   71267128  +2 
   ==
   + Hits  34200   34271 +71 
   + Misses14502   14455 -47 
   + Partials   23452344  -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...elix/core/relocation/RealtimeSegmentRelocator.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlbG9jYXRpb24vUmVhbHRpbWVTZWdtZW50UmVsb2NhdG9yLmphdmE=)
 | `73.62% <0%> (-7.7%)` | `0% <0%> (ø)` | |
   | 
[.../apache/pinot/common/config/RealtimeTagConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1JlYWx0aW1lVGFnQ29uZmlnLmphdmE=)
 | `93.33% <0%> (-6.67%)` | `0% <0%> (ø)` | |
   | 
[...impl/dictionary/FloatOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRmxvYXRPZmZIZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==)
 | `87.27% <0%> (-5.46%)` | `0% <0%> (ø)` | |
   | 
[...a/org/apache/pinot/common/config/TagNameUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1RhZ05hbWVVdGlscy5qYXZh)
 | `64.1% <0%> (-5.13%)` | `0% <0%> (ø)` | |
   | 
[...mpl/dictionary/DoubleOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT2ZmSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=)
 | `72.72% <0%> (-3.64%)` | `0% <0%> (ø)` | |
   | 
[...g/apache/pinot/common/metrics/AbstractMetrics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9BYnN0cmFjdE1ldHJpY3MuamF2YQ==)
 | `75.33% <0%> (-1.34%)` | `0% <0%> (ø)` | |
   | 
[...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh)
 | `71.85% <0%> (-1.01%)` | `0% <0%> (ø)` | |
   | 
[.../impl/dictionary/FloatOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRmxvYXRPbkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh)
 | `86.66% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
   | 
[...altime/impl/kafka/KafkaStreamConfigProperties.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2thZmthL0thZmthU3RyZWFtQ29uZmlnUHJvcGVydGllcy5qYXZh)
 | `25% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: |
   | 
[...re/realtime/impl/kafka/KafkaConnectionHandler.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2thZmthL0thZmthQ29ubmVjdGlvbkhhbmRsZXIuamF2YQ==)
 | `70.05% <0%> (+0.18%)` | `0% <0%> (ø)` | :arrow_down: |
   | ... and [15 
more](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=continue).
   > 

[GitHub] [incubator-pinot] codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table Configs, Monitoring, and Deployment

[GitBox]

codecov-io edited a comment on issue #3975: ReadTheDocs documentation for Table 
Configs, Monitoring, and Deployment
URL: https://github.com/apache/incubator-pinot/pull/3975#issuecomment-473083001
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=h1) 
Report
   > Merging 
[#3975](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-pinot/commit/5867b16f32145a80fa98de4d24ec6302b17bb95e?src=pr=desc)
 will **increase** coverage by `0.25%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-pinot/pull/3975/graphs/tree.svg?width=650=4ibza2ugkz=150=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#3975  +/-   ##
   
   + Coverage 66.99%   67.25%   +0.25% 
 Complexity44  
   
 Files  1032 1032  
 Lines 5104751070  +23 
 Branches   7126 7128   +2 
   
   + Hits  3420034347 +147 
   + Misses1450214373 -129 
   - Partials   2345 2350   +5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-pinot/pull/3975?src=pr=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==)
 | `75% <0%> (-25%)` | `0% <0%> (ø)` | |
   | 
[...elix/core/relocation/RealtimeSegmentRelocator.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlbG9jYXRpb24vUmVhbHRpbWVTZWdtZW50UmVsb2NhdG9yLmphdmE=)
 | `73.62% <0%> (-7.7%)` | `0% <0%> (ø)` | |
   | 
[...mpl/dictionary/DoubleOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT2ZmSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=)
 | `69.09% <0%> (-7.28%)` | `0% <0%> (ø)` | |
   | 
[.../impl/dictionary/LongOffHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09mZkhlYXBNdXRhYmxlRGljdGlvbmFyeS5qYXZh)
 | `85.45% <0%> (-7.28%)` | `0% <0%> (ø)` | |
   | 
[...e/impl/dictionary/LongOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09uSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=)
 | `82.22% <0%> (-6.67%)` | `0% <0%> (ø)` | |
   | 
[.../apache/pinot/common/config/RealtimeTagConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1JlYWx0aW1lVGFnQ29uZmlnLmphdmE=)
 | `93.33% <0%> (-6.67%)` | `0% <0%> (ø)` | |
   | 
[...g/apache/pinot/common/metrics/AbstractMetrics.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9BYnN0cmFjdE1ldHJpY3MuamF2YQ==)
 | `75.33% <0%> (-1.34%)` | `0% <0%> (ø)` | |
   | 
[...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh)
 | `71.85% <0%> (-1.01%)` | `0% <0%> (ø)` | |
   | 
[...pby/NoDictionarySingleColumnGroupKeyGenerator.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9ncm91cGJ5L05vRGljdGlvbmFyeVNpbmdsZUNvbHVtbkdyb3VwS2V5R2VuZXJhdG9yLmphdmE=)
 | `87.5% <0%> (-0.97%)` | `0% <0%> (ø)` | |
   | 
[...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==)
 | `78.62% <0%> (-0.77%)` | `0% <0%> (ø)` | |
   | ... and [27 
more](https://codecov.io/gh/apache/incubator-pinot/pull/3975/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 

[GitHub] [incubator-pinot] jihaozh merged pull request #3981: [TE] detection - align metric slices

[GitBox]

jihaozh merged pull request #3981: [TE] detection - align metric slices
URL: https://github.com/apache/incubator-pinot/pull/3981
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: [TE] detection - align metric slices (#3981)

[jihao]

This is an automated email from the ASF dual-hosted git repository.

jihao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new f815e2e  [TE] detection - align metric slices (#3981)
f815e2e is described below

commit f815e2ea569dfe63d74e69e8dff69fe2f5a2b559
Author: Jihao Zhang 
AuthorDate: Mon Mar 18 17:30:32 2019 -0700

[TE] detection - align metric slices (#3981)

Align metric slices to the data granularity.
---
 .../thirdeye/detection/DefaultDataProvider.java| 42 --
 .../detection/wrapper/AnomalyDetectorWrapper.java  |  2 +-
 2 files changed, 41 insertions(+), 3 deletions(-)

diff --git 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DefaultDataProvider.java
 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DefaultDataProvider.java
index a7e9026..935b629 100644
--- 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DefaultDataProvider.java
+++ 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DefaultDataProvider.java
@@ -39,6 +39,8 @@ import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import org.apache.pinot.thirdeye.common.time.TimeGranularity;
 import org.apache.pinot.thirdeye.dataframe.DataFrame;
 import org.apache.pinot.thirdeye.dataframe.util.MetricSlice;
 import org.apache.pinot.thirdeye.datalayer.bao.DatasetConfigManager;
@@ -51,10 +53,12 @@ import org.apache.pinot.thirdeye.datalayer.dto.EventDTO;
 import org.apache.pinot.thirdeye.datalayer.dto.MergedAnomalyResultDTO;
 import org.apache.pinot.thirdeye.datalayer.dto.MetricConfigDTO;
 import org.apache.pinot.thirdeye.datalayer.util.Predicate;
+import org.apache.pinot.thirdeye.datasource.comparison.Row;
 import org.apache.pinot.thirdeye.datasource.loader.AggregationLoader;
 import org.apache.pinot.thirdeye.datasource.loader.TimeSeriesLoader;
 import org.apache.pinot.thirdeye.detection.spi.model.AnomalySlice;
 import org.apache.pinot.thirdeye.detection.spi.model.EventSlice;
+import org.joda.time.DateTimeZone;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -154,11 +158,15 @@ public class DefaultDataProvider implements DataProvider {
   @Override
   public Map fetchTimeseries(Collection 
slices) {
 try {
-  Map cacheResult = 
DETECTION_TIME_SERIES_CACHE.getAll(slices);
+  Map alignedMetricSlicesToOriginalSlice = new 
HashMap<>();
+  for (MetricSlice slice: slices) {
+alignedMetricSlicesToOriginalSlice.put(alignSlice(slice), slice);
+  }
+  Map cacheResult = 
DETECTION_TIME_SERIES_CACHE.getAll(alignedMetricSlicesToOriginalSlice.keySet());
   Map timeseriesResult = new HashMap<>();
   for (Map.Entry entry : cacheResult.entrySet()){
 // make a copy of the result so that cache won't be contaminated by 
client code
-timeseriesResult.put(entry.getKey(), entry.getValue().copy());
+
timeseriesResult.put(alignedMetricSlicesToOriginalSlice.get(entry.getKey()), 
entry.getValue().copy());
   }
   return  timeseriesResult;
 } catch (Exception e) {
@@ -314,6 +322,36 @@ public class DefaultDataProvider implements DataProvider {
 return diff > 0 ? diff : 0;
   }
 
+  /**
+   * Aligns a metric slice based on its granularity, or the dataset 
granularity.
+   *
+   * @param slice metric slice
+   * @return aligned metric slice
+   */
+  private MetricSlice alignSlice(MetricSlice slice) {
+MetricConfigDTO metric = this.metricDAO.findById(slice.getMetricId());
+if (metric == null) {
+  throw new IllegalArgumentException(String.format("Could not resolve 
metric id %d", slice.getMetricId()));
+}
+
+DatasetConfigDTO dataset = 
this.datasetDAO.findByDataset(metric.getDataset());
+if (dataset == null) {
+  throw new IllegalArgumentException(String.format("Could not resolve 
dataset '%s' for metric id %d", metric.getDataset(), slice.getMetricId()));
+}
+
+TimeGranularity granularity = dataset.bucketTimeGranularity();
+if (!MetricSlice.NATIVE_GRANULARITY.equals(slice.getGranularity())) {
+  granularity = slice.getGranularity();
+}
+
+// align to time buckets and request time zone
+long timeGranularity = granularity.toMillis();
+long start = (slice.getStart() / timeGranularity) * timeGranularity;
+long end = ((slice.getEnd() + timeGranularity - 1) / timeGranularity) * 
timeGranularity;
+
+return slice.withStart(start).withEnd(end).withGranularity(granularity);
+  }
+
   public static void cleanCache() {
 if (DETECTION_TIME_SERIES_CACHE != null) {
   DETECTION_TIME_SERIES_CACHE.cleanUp();
diff --git 

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #3854: Upgrade Helix version to 0.8.3

[GitBox]

Jackie-Jiang commented on issue #3854: Upgrade Helix version to 0.8.3
URL: https://github.com/apache/incubator-pinot/pull/3854#issuecomment-474150389
 
 
   Closed because Helix issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang closed pull request #3854: Upgrade Helix version to 0.8.3

[GitBox]

Jackie-Jiang closed pull request #3854: Upgrade Helix version to 0.8.3
URL: https://github.com/apache/incubator-pinot/pull/3854
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] snleee merged pull request #3869: add config to control kafka fetcher size and increase default

[GitBox]

snleee merged pull request #3869: add config to control kafka fetcher size and 
increase default
URL: https://github.com/apache/incubator-pinot/pull/3869
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: add config to control kafka fetcher size and increase default (#3869)

[snlee]

This is an automated email from the ASF dual-hosted git repository.

snlee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 565171f  add config to control kafka fetcher size and increase default 
(#3869)
565171f is described below

commit 565171f23cc4af4e5a2c68b0d64194561a34ea9a
Author: James Shao 
AuthorDate: Mon Mar 18 17:08:32 2019 -0700

add config to control kafka fetcher size and increase default (#3869)

* add config to control kafka fetcher size and increase default

* update per cr feedback

* change default back to old value
---
 .../impl/kafka/KafkaConnectionHandler.java | 26 +++
 .../impl/kafka/KafkaLowLevelStreamConfig.java  | 34 +++---
 .../impl/kafka/KafkaPartitionLevelConsumer.java| 21 -
 .../impl/kafka/KafkaStreamConfigProperties.java|  3 ++
 .../impl/kafka/KafkaLowLevelStreamConfigTest.java  | 53 +-
 .../kafka/KafkaPartitionLevelConsumerTest.java |  7 +++
 6 files changed, 124 insertions(+), 20 deletions(-)

diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaConnectionHandler.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaConnectionHandler.java
index 23c5528..a3366a3 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaConnectionHandler.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaConnectionHandler.java
@@ -77,6 +77,8 @@ public class KafkaConnectionHandler {
 
   boolean isPartitionProvided;
 
+  private final KafkaLowLevelStreamConfig _kafkaLowLevelStreamConfig;
+
   @VisibleForTesting
   public SimpleConsumer getSimpleConsumer() {
 return _simpleConsumer;
@@ -109,19 +111,19 @@ public class KafkaConnectionHandler {
*/
   public KafkaConnectionHandler(String clientId, StreamConfig streamConfig,
   KafkaSimpleConsumerFactory simpleConsumerFactory) {
-KafkaLowLevelStreamConfig kafkaLowLevelStreamConfig = new 
KafkaLowLevelStreamConfig(streamConfig);
+_kafkaLowLevelStreamConfig = new KafkaLowLevelStreamConfig(streamConfig);
 _simpleConsumerFactory = simpleConsumerFactory;
 _clientId = clientId;
-_topic = kafkaLowLevelStreamConfig.getKafkaTopicName();
+_topic = _kafkaLowLevelStreamConfig.getKafkaTopicName();
 _connectTimeoutMillis = streamConfig.getConnectionTimeoutMillis();
 _simpleConsumer = null;
 
 isPartitionProvided = false;
 _partition = Integer.MIN_VALUE;
 
-_bufferSize = kafkaLowLevelStreamConfig.getKafkaBufferSize();
-_socketTimeout = kafkaLowLevelStreamConfig.getKafkaSocketTimeout();
-initializeBootstrapNodeList(kafkaLowLevelStreamConfig.getBootstrapHosts());
+_bufferSize = _kafkaLowLevelStreamConfig.getKafkaBufferSize();
+_socketTimeout = _kafkaLowLevelStreamConfig.getKafkaSocketTimeout();
+
initializeBootstrapNodeList(_kafkaLowLevelStreamConfig.getBootstrapHosts());
 setCurrentState(new ConnectingToBootstrapNode());
   }
 
@@ -134,19 +136,19 @@ public class KafkaConnectionHandler {
   public KafkaConnectionHandler(String clientId, StreamConfig streamConfig, 
int partition,
   KafkaSimpleConsumerFactory simpleConsumerFactory) {
 
-KafkaLowLevelStreamConfig kafkaLowLevelStreamConfig = new 
KafkaLowLevelStreamConfig(streamConfig);
+_kafkaLowLevelStreamConfig = new KafkaLowLevelStreamConfig(streamConfig);
 _simpleConsumerFactory = simpleConsumerFactory;
 _clientId = clientId;
-_topic = kafkaLowLevelStreamConfig.getKafkaTopicName();
+_topic = _kafkaLowLevelStreamConfig.getKafkaTopicName();
 _connectTimeoutMillis = streamConfig.getConnectionTimeoutMillis();
 _simpleConsumer = null;
 
 isPartitionProvided = true;
 _partition = partition;
 
-_bufferSize = kafkaLowLevelStreamConfig.getKafkaBufferSize();
-_socketTimeout = kafkaLowLevelStreamConfig.getKafkaSocketTimeout();
-initializeBootstrapNodeList(kafkaLowLevelStreamConfig.getBootstrapHosts());
+_bufferSize = _kafkaLowLevelStreamConfig.getKafkaBufferSize();
+_socketTimeout = _kafkaLowLevelStreamConfig.getKafkaSocketTimeout();
+
initializeBootstrapNodeList(_kafkaLowLevelStreamConfig.getBootstrapHosts());
 setCurrentState(new ConnectingToBootstrapNode());
   }
 
@@ -180,6 +182,10 @@ public class KafkaConnectionHandler {
 }
   }
 
+  protected KafkaLowLevelStreamConfig getkafkaLowLevelStreamConfig() {
+return _kafkaLowLevelStreamConfig;
+  }
+
   abstract class State {
 private ConsumerState stateValue;
 
diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaLowLevelStreamConfig.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/kafka/KafkaLowLevelStreamConfig.java
index 76a5287..00ed1fd 100644
--- 

[GitHub] [incubator-pinot] jihaozh merged pull request #3983: [TE] detection - preview a yaml with existing anomalies

[GitBox]

jihaozh merged pull request #3983: [TE] detection - preview a yaml with 
existing anomalies
URL: https://github.com/apache/incubator-pinot/pull/3983
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: [TE] detection - preview a yaml with existing anomalies (#3983)

[jihao]

This is an automated email from the ASF dual-hosted git repository.

jihao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 9e8e373  [TE] detection - preview a yaml with existing anomalies 
(#3983)
9e8e373 is described below

commit 9e8e3731cd6cc356a421fc3f27b34cba15f1afc8
Author: Jihao Zhang 
AuthorDate: Mon Mar 18 16:23:22 2019 -0700

[TE] detection - preview a yaml with existing anomalies (#3983)

Preview a YAML with existing anomalies.
---
 .../thirdeye/detection/DetectionResource.java  |  5 +++
 .../thirdeye/detection/yaml/YamlResource.java  | 40 ++
 2 files changed, 45 insertions(+)

diff --git 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
index 0b3146a..894b2ac 100644
--- 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
+++ 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
@@ -177,6 +177,7 @@ public class DetectionResource {
 
   @POST
   @Path("/preview")
+  @ApiOperation("preview a detection with detection config json")
   public Response detectionPreview(
   @QueryParam("start") long start,
   @QueryParam("end") long end,
@@ -230,6 +231,7 @@ public class DetectionResource {
 
   @POST
   @Path("/preview/{id}")
+  @ApiOperation("preview a detection with a existing detection config")
   public Response detectionPreview(
   @PathParam("id") long id,
   @QueryParam("start") long start,
@@ -340,6 +342,8 @@ public class DetectionResource {
*/
   @POST
   @Path("/legacy-replay/{id}")
+  @ApiOperation("Legacy replay endpoint. Replay all the moving windows within 
start time and end time. "
+  + "Saves anomaly for each moving window before starting detection for 
next window and emulates the cron schedule")
   public Response legacyReplay(
   @PathParam("id") long configId,
   @QueryParam("start") long start,
@@ -400,6 +404,7 @@ public class DetectionResource {
*/
   @POST
   @Path("/replay/{id}")
+  @ApiOperation("Replay for a given time range for a existing detection config 
id")
   public Response detectionReplay(
   @PathParam("id") long detectionId,
   @QueryParam("start") long start,
diff --git 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/YamlResource.java
 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/YamlResource.java
index 67e7327..98945a5 100644
--- 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/YamlResource.java
+++ 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/YamlResource.java
@@ -586,6 +586,46 @@ public class YamlResource {
 return Response.ok(result).build();
   }
 
+  @POST
+  @Path("/preview/{id}")
+  @Produces(MediaType.APPLICATION_JSON)
+  @Consumes(MediaType.TEXT_PLAIN)
+  @ApiOperation("Preview the anomaly detection result of a YAML configuration, 
with an existing config and historical anomalies")
+  public Response yamlPreviewWithHistoricalAnomalies(
+  @PathParam("id") long id,
+  @QueryParam("start") long start,
+  @QueryParam("end") long end,
+  @QueryParam("tuningStart") long tuningStart,
+  @QueryParam("tuningEnd") long tuningEnd,
+  @ApiParam("jsonPayload") String payload) {
+Map responseMessage = new HashMap<>();
+DetectionPipelineResult result;
+long ts = System.currentTimeMillis();
+try {
+  Preconditions.checkArgument(StringUtils.isNotBlank(payload), "The Yaml 
Payload in the request is empty.");
+  DetectionConfigDTO existingConfig = this.detectionConfigDAO.findById(id);
+  Preconditions.checkNotNull(existingConfig, "can not find existing 
detection config " + id);
+
+  // Translate config from YAML to detection config (JSON)
+  Map newDetectionConfigMap = new 
HashMap<>(ConfigUtils.getMap(this.yaml.load(payload)));
+  DetectionConfigDTO detectionConfig = 
buildDetectionConfigFromYaml(tuningStart, tuningEnd, newDetectionConfigMap, 
existingConfig);
+  Preconditions.checkNotNull(detectionConfig);
+
+  DetectionPipeline pipeline = this.loader.from(this.provider, 
detectionConfig, start, end);
+  result = pipeline.run();
+
+} catch (ValidationException e) {
+  LOG.warn("Validation error while running preview with payload  " + 
payload, e);
+  responseMessage.put("message", "Validation Error! " + e.getMessage());
+  return Response.serverError().entity(responseMessage).build();
+} catch (Exception e) {
+  LOG.error("Error running preview with payload " + payload, e);
+  responseMessage.put("message", "Failed to run the preview due to " + 
e.getMessage());
+  

[incubator-pinot] branch orc updated (ffed46d -> 4a3ce08)

[jenniferdai]

This is an automated email from the ASF dual-hosted git repository.

jenniferdai pushed a change to branch orc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


 discard ffed46d  ORC Reader
 discard d851c30  changing to version
 discard dddf153  Updating 0.016 to 0.1.0 for pinot-hadoop and docs
 new 6e541bb  Fix Preconditions check (#3965)
 new 2411d89  Update dev guide to include toc for high level sections 
(#3973)
 new 9ded2a6  Add more documents in Admin Guide (#3972)
 new 5867b16  Add documentation for tuning scatter and gather (#3969)
 new b8c3ff4  Updating 0.016 to 0.1.0 for pinot-hadoop and docs (#3971)
 new 7b0b00c  [TE] Fix alert filter and enrich header (#3974)
 new b2c5a88  Make RetentionManager and OfflineSegmentIntervalChecker 
initial delays configurable (#3946)
 new 581464e  Spelling correction (#3977)
 new 636c6c1  [TE] frontend - harleyjj/home - set default date picker to 
yesterday (#3976)
 new 67b729d  [TE] yaml - more validation on max duration (#3982)
 new 4a3ce08  Adding orc reader

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (ffed46d)
\
 N -- N -- N   refs/heads/orc (4a3ce08)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 5849 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 docs/client_api.rst|   2 +-
 docs/code_modules.rst  |   6 +-
 docs/contribution_guidelines.rst   |  19 ++-
 docs/img/generate-segment.png  | Bin 0 -> 218597 bytes
 docs/img/list-schemas.png  | Bin 0 -> 8952 bytes
 docs/img/partitioning.png  | Bin 0 -> 81303 bytes
 docs/img/query-table.png   | Bin 0 -> 35914 bytes
 docs/img/replica-group.png | Bin 0 -> 126306 bytes
 docs/img/upload-segment.png| Bin 0 -> 13944 bytes
 docs/index_techniques.rst  |  35 ++--
 docs/management_api.rst|  32 +++-
 docs/pinot_hadoop.rst  |  71 +++-
 docs/tuning_pinot.rst  |  12 +-
 docs/tuning_scatter_and_gather.rst | 182 +
 .../apache/pinot/controller/ControllerConf.java|  14 ++
 .../generator/ConvertToRawIndexTaskGenerator.java  |   2 +-
 .../helix/core/retention/RetentionManager.java |   2 +-
 .../validation/OfflineSegmentIntervalChecker.java  |   2 +-
 .../app/mirage/factories/alert.js  |   2 +-
 .../self-serve-alert-details/template.hbs  |   4 +-
 .../app/pods/home/index/controller.js  |   8 +-
 .../thirdeye-frontend/app/pods/home/index/route.js |  12 +-
 .../app/pods/manage/alerts/index/controller.js |   6 +-
 .../app/pods/manage/alerts/index/route.js  |  20 +--
 .../app/pods/manage/explore/route.js   |   2 +-
 .../app/pods/manage/yaml/route.js  |  67 +---
 .../app/pods/manage/yaml/template.hbs  |   3 +-
 .../yaml/CompositePipelineConfigTranslator.java|  29 ++--
 .../thirdeye/detection/yaml/YamlResource.java  |  11 +-
 29 files changed, 434 insertions(+), 109 deletions(-)
 create mode 100644 docs/img/generate-segment.png
 create mode 100644 docs/img/list-schemas.png
 create mode 100644 docs/img/partitioning.png
 create mode 100644 docs/img/query-table.png
 create mode 100644 docs/img/replica-group.png
 create mode 100644 docs/img/upload-segment.png
 create mode 100644 docs/tuning_scatter_and_gather.rst


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] 01/01: ORC Reader

[jenniferdai]

This is an automated email from the ASF dual-hosted git repository.

jenniferdai pushed a commit to branch orc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git

commit ffed46d2a333dd64722565dc98c16b4965b5a96e
Author: Jennifer Dai 
AuthorDate: Mon Mar 18 15:03:41 2019 -0700

ORC Reader
---
 .../java/org/apache/pinot/common/data/Schema.java  |   1 +
 pinot-orc/pom.xml  |  58 ++
 .../pinot/orc/data/readers/ORCRecordReader.java| 126 +
 pom.xml|  11 ++
 4 files changed, 196 insertions(+)

diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/data/Schema.java 
b/pinot-common/src/main/java/org/apache/pinot/common/data/Schema.java
index f06b14c..087c582 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/data/Schema.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/data/Schema.java
@@ -138,6 +138,7 @@ public final class Schema {
 return _metricFieldSpecs;
   }
 
+
   /**
* Required by JSON deserializer. DO NOT USE. DO NOT REMOVE.
* Adding @Deprecated to prevent usage
diff --git a/pinot-orc/pom.xml b/pinot-orc/pom.xml
new file mode 100644
index 000..10c0a18
--- /dev/null
+++ b/pinot-orc/pom.xml
@@ -0,0 +1,58 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  
+pinot
+org.apache.pinot
+0.1.0-SNAPSHOT
+  
+  4.0.0
+
+  pinot-orc
+  Pinot ORC
+  https://pinot.apache.org/
+  
+${basedir}/..
+  
+  
+
+  
+org.apache.maven.plugins
+maven-enforcer-plugin
+  
+
+  
+  
+
+  org.apache.orc
+  orc-core
+
+
+  org.apache.orc
+  orc-mapreduce
+
+
+  org.apache.pinot
+  pinot-core
+
+  
+
\ No newline at end of file
diff --git 
a/pinot-orc/src/main/java/org/apache/pinot/orc/data/readers/ORCRecordReader.java
 
b/pinot-orc/src/main/java/org/apache/pinot/orc/data/readers/ORCRecordReader.java
new file mode 100644
index 000..68660fd
--- /dev/null
+++ 
b/pinot-orc/src/main/java/org/apache/pinot/orc/data/readers/ORCRecordReader.java
@@ -0,0 +1,126 @@
+package org.apache.pinot.orc.data.readers;
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.orc.OrcFile;
+import org.apache.orc.Reader;
+import org.apache.orc.TypeDescription;
+import org.apache.pinot.common.data.Schema;
+import org.apache.pinot.core.data.GenericRow;
+import org.apache.pinot.core.data.readers.RecordReader;
+import org.apache.pinot.core.indexsegment.generator.SegmentGeneratorConfig;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class ORCRecordReader implements RecordReader {
+
+  private Schema _pinotSchema;
+  private TypeDescription _orcSchema;
+  Reader _reader;
+  org.apache.orc.RecordReader _recordReader;
+  VectorizedRowBatch _reusableVectorizedRowBatch = new VectorizedRowBatch(1);
+
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(ORCRecordReader.class);
+
+  @Override
+  public void init(SegmentGeneratorConfig segmentGeneratorConfig) {
+Configuration conf = new Configuration();
+LOGGER.info("Creating segment for {}", 
segmentGeneratorConfig.getInputFilePath());
+try {
+  _reader = OrcFile.createReader(new 
Path(segmentGeneratorConfig.getInputFilePath()), OrcFile.readerOptions(conf));
+  _orcSchema = _reader.getSchema();
+  LOGGER.info("ORC schema is {}", _orcSchema.toJson());
+
+  _pinotSchema = segmentGeneratorConfig.getSchema();
+  if (_pinotSchema == null) {
+throw new IllegalArgumentException("ORCRecordReader requires schema");
+  }
+  _recordReader = _reader.rows(_reader.options().schema(_orcSchema));
+} catch (Exception e) {
+  LOGGER.error("Caught exception initializing record reader at path {}", 
segmentGeneratorConfig.getInputFilePath());
+  throw new 

[incubator-pinot] branch orc created (now ffed46d)

[jenniferdai]

This is an automated email from the ASF dual-hosted git repository.

jenniferdai pushed a change to branch orc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


  at ffed46d  ORC Reader

This branch includes the following new commits:

 new ffed46d  ORC Reader

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jihaozh opened a new pull request #3983: [TE] detection - preview a yaml with existing anomalies

[GitBox]

jihaozh opened a new pull request #3983: [TE] detection - preview a yaml with 
existing anomalies
URL: https://github.com/apache/incubator-pinot/pull/3983
 
 
   Preview a YAML with existing anomalies.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.

[GitBox]

chenboat commented on a change in pull request #3941: Pinot server side change 
to optimize LLC segment completion with direct metadata upload. 
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r266643302
 
 

 ##
 File path: 
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
 ##
 @@ -735,14 +742,45 @@ protected SegmentBuildDescriptor 
buildSegmentInternal(boolean forCommit) {
 if (_isOffHeap) {
   params.withMemoryUsedBytes(_memoryManager.getTotalAllocatedBytes());
 }
-SegmentCompletionProtocol.Response commitEndResponse = 
_protocolHandler.segmentCommitEnd(params);
+SegmentCompletionProtocol.Response commitEndResponse;
+if (!_indexLoadingConfig.isEnableSplitCommitEndWithMetadata()) {
+  commitEndResponse = _protocolHandler.segmentCommitEnd(params);
+} else {
+  Map metadataFiles = new HashMap<>();
+  try {
+metadataFiles.put(V1Constants.MetadataKeys.METADATA_FILE_NAME,
 
 Review comment:
   I see. Then we can just use the existing metadata files on the server -- and 
there is no need to delete any temp files. I tested it and it works. Please 
check the latest commits.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.

[GitBox]

chenboat commented on a change in pull request #3941: Pinot server side change 
to optimize LLC segment completion with direct metadata upload. 
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r266642680
 
 

 ##
 File path: 
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManagerConfig.java
 ##
 @@ -163,6 +165,11 @@ public boolean isEnableSplitCommit() {
 return _instanceDataManagerConfiguration.getBoolean(ENABLE_SPLIT_COMMIT, 
false);
   }
 
+  @Override
+  public boolean isEnableSplitCommitEndWithMetadata() {
+return 
_instanceDataManagerConfiguration.getBoolean(ENABLE_SPLIT_COMMIT_END_WITH_METADATA,
 false);
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on issue #3951: Upgrade to Helix version 0.8.4.

[GitBox]

mcvsubbu commented on issue #3951: Upgrade to Helix version 0.8.4.
URL: https://github.com/apache/incubator-pinot/pull/3951#issuecomment-474088149
 
 
   Please keep the branch since it is being used for debugging


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] 01/01: Add documentation

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch add-docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git

commit 5176b8237ac5832399e295a716db47ffc65d6aca
Author: jackjlli 
AuthorDate: Thu Mar 14 17:18:31 2019 -0700

Add documentation
---
 docs/getting_started.rst   |   7 +++
 docs/img/pinot-console.png | Bin 0 -> 157310 bytes
 docs/management_api.rst|  49 +++--
 3 files changed, 54 insertions(+), 2 deletions(-)

diff --git a/docs/getting_started.rst b/docs/getting_started.rst
index e3d2416..4b92f96 100644
--- a/docs/getting_started.rst
+++ b/docs/getting_started.rst
@@ -94,3 +94,10 @@ show up in Pinot.
 To show new events appearing, one can run :sql:`SELECT * FROM meetupRsvp ORDER 
BY mtime DESC LIMIT 50` repeatedly, which shows the
 last events that were ingested by Pinot.
 
+Experiment with Pinot
+~
+
+Here's the instruction on how to add a simple table to the pinot system you 
got working int the previous demo steps,
+how to add segments, and how to query it.
+
+
diff --git a/docs/img/pinot-console.png b/docs/img/pinot-console.png
new file mode 100644
index 000..f73405b
Binary files /dev/null and b/docs/img/pinot-console.png differ
diff --git a/docs/management_api.rst b/docs/management_api.rst
index 5a7a0e4..3aaef33 100644
--- a/docs/management_api.rst
+++ b/docs/management_api.rst
@@ -28,6 +28,8 @@ Pinot Management Console
 There is a REST API which allows management of tables, tenants, segments and 
schemas. It can be accessed by going to
 ``http://[controller host]/help`` which offers a web UI to do these tasks, as 
well as document the REST API.
 
+  .. figure:: img/pinot-console.png
+
 For example, list all the schema within Pinot cluster:
 
   .. figure:: img/list-schemas.png
@@ -44,8 +46,51 @@ It can be used instead of the ``pinot-admin.sh`` commands to 
automate the creati
 
 For example, create a pinot segment:
 
-  .. figure:: img/generate-segment.png
+.. code-block:: none
+
+  $ 
./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh
 CreateSegment -dataDir /Users/jlli/Desktop/test/ -format CSV -outDir 
/Users/jlli/Desktop/test2/ -tableName baseballStats -segmentName 
baseballStats_data -overwrite -schemaFile 
./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json
+  Executing command: CreateSegment  -generatorConfigFile null -dataDir 
/Users/jlli/Desktop/test/ -format CSV -outDir /Users/jlli/Desktop/test2/ 
-overwrite true -tableName baseballStats -segmentName baseballStats_data 
-timeColumnName null -schemaFile 
./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json
 -readerConfigFile null -enableStarTreeIndex false -starTreeIndexSpecFile null 
-hllSize 9 - [...]
+  Accepted files: [/Users/jlli/Desktop/test/baseballStats_data.csv]
+  Finished building StatsCollector!
+  Collected stats for 97889 documents
+  Created dictionary for INT column: homeRuns with cardinality: 67, range: 0 
to 73
+  Created dictionary for INT column: playerStint with cardinality: 5, range: 1 
to 5
+  Created dictionary for INT column: groundedIntoDoublePlays with cardinality: 
35, range: 0 to 36
+  Created dictionary for INT column: numberOfGames with cardinality: 165, 
range: 1 to 165
+  Created dictionary for INT column: AtBatting with cardinality: 699, range: 0 
to 716
+  Created dictionary for INT column: stolenBases with cardinality: 114, range: 
0 to 138
+  Created dictionary for INT column: tripples with cardinality: 32, range: 0 
to 36
+  Created dictionary for INT column: hitsByPitch with cardinality: 41, range: 
0 to 51
+  Created dictionary for STRING column: teamID with cardinality: 149, max 
length in bytes: 3, range: ALT to WSU
+  Created dictionary for INT column: numberOfGamesAsBatter with cardinality: 
166, range: 0 to 165
+  Created dictionary for INT column: strikeouts with cardinality: 199, range: 
0 to 223
+  Created dictionary for INT column: sacrificeFlies with cardinality: 20, 
range: 0 to 19
+  Created dictionary for INT column: caughtStealing with cardinality: 36, 
range: 0 to 42
+  Created dictionary for INT column: baseOnBalls with cardinality: 154, range: 
0 to 232
+  Created dictionary for STRING column: playerName with cardinality: 11976, 
max length in bytes: 43, range:  to Zoilo Casanova
+  Created dictionary for INT column: doules with cardinality: 64, range: 0 to 
67
+  Created dictionary for STRING column: league with cardinality: 7, max length 
in bytes: 2, range: AA to UA
+  Created dictionary for INT column: yearID with cardinality: 143, range: 1871 
to 2013
+  Created dictionary for INT column: hits with cardinality: 250, range: 0 to 
262
+  Created dictionary 

[incubator-pinot] branch add-docs created (now 5176b82)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch add-docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.


  at 5176b82  Add documentation

This branch includes the following new commits:

 new 5176b82  Add documentation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mayankshriv commented on issue #3951: Upgrade to Helix version 0.8.4.

[GitBox]

mayankshriv commented on issue #3951: Upgrade to Helix version 0.8.4.
URL: https://github.com/apache/incubator-pinot/pull/3951#issuecomment-474076158
 
 
   The Helix version 0.8.4 turns out to be unstable, specifically, the helix 
task manager seems to have slowed down quite a bit. Will upgrade to a new helix 
version with the fixes when available, closing the pull request.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mayankshriv closed pull request #3951: Upgrade to Helix version 0.8.4.

[GitBox]

mayankshriv closed pull request #3951: Upgrade to Helix version 0.8.4.
URL: https://github.com/apache/incubator-pinot/pull/3951
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: [TE] yaml - more validation on max duration (#3982)

[jihao]

This is an automated email from the ASF dual-hosted git repository.

jihao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 67b729d  [TE] yaml - more validation on max duration (#3982)
67b729d is described below

commit 67b729d4b0935e652be31d5a3fff025b9f361204
Author: Jihao Zhang 
AuthorDate: Mon Mar 18 12:28:26 2019 -0700

[TE] yaml - more validation on max duration (#3982)
---
 .../yaml/CompositePipelineConfigTranslator.java| 29 +++---
 1 file changed, 14 insertions(+), 15 deletions(-)

diff --git 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/CompositePipelineConfigTranslator.java
 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/CompositePipelineConfigTranslator.java
index 13ffbdf..657efa6 100644
--- 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/CompositePipelineConfigTranslator.java
+++ 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/CompositePipelineConfigTranslator.java
@@ -182,19 +182,17 @@ public class CompositePipelineConfigTranslator extends 
YamlDetectionConfigTransl
   private static final Set MOVING_WINDOW_DETECTOR_TYPES = 
ImmutableSet.of("ALGORITHM", "MIGRATED_ALGORITHM");
 
   private final Map components = new HashMap<>();
-  private MetricConfigDTO metricConfig;
-  private DatasetConfigDTO datasetConfig;
-  private String metricUrn;
-  private Map mergerProperties = new HashMap<>();
+  private final MetricConfigDTO metricConfig;
+  private final DatasetConfigDTO datasetConfig;
+  private final String metricUrn;
+  private final Map mergerProperties;
+  // metric dimension filter maps
+  private final Map> filterMaps;
   protected final org.yaml.snakeyaml.Yaml yaml;
 
   public CompositePipelineConfigTranslator(Map yamlConfig, 
DataProvider provider) {
 super(yamlConfig, provider);
 this.yaml = new org.yaml.snakeyaml.Yaml();
-  }
-
-  @Override
-  YamlTranslationResult translateYaml() {
 this.metricConfig = 
this.dataProvider.fetchMetric(MapUtils.getString(yamlConfig, PROP_METRIC),
 MapUtils.getString(yamlConfig, PROP_DATASET));
 Preconditions.checkNotNull(this.metricConfig, "Metric not found");
@@ -202,12 +200,13 @@ public class CompositePipelineConfigTranslator extends 
YamlDetectionConfigTransl
 this.datasetConfig = 
this.dataProvider.fetchDatasets(Collections.singletonList(metricConfig.getDataset()))
 .get(metricConfig.getDataset());
 Preconditions.checkNotNull(this.datasetConfig, "dataset not found");
-
-// if user set merger properties
 this.mergerProperties = MapUtils.getMap(yamlConfig, PROP_MERGER, new 
HashMap());
-
-Map> filterMaps = MapUtils.getMap(yamlConfig, 
PROP_FILTERS);
+this.filterMaps = MapUtils.getMap(yamlConfig, PROP_FILTERS);
 this.metricUrn = buildMetricUrn(filterMaps, this.metricConfig.getId());
+  }
+
+  @Override
+  YamlTranslationResult translateYaml() {
 String detectionCronInYaml = MapUtils.getString(yamlConfig, PROP_CRON);
 String cron = (detectionCronInYaml == null) ? buildCron() : 
detectionCronInYaml;
 
@@ -228,14 +227,14 @@ public class CompositePipelineConfigTranslator extends 
YamlDetectionConfigTransl
 nestedPipelines.addAll(filterNestedProperties);
   }
 }
-Map dimensionWrapperProperties = 
buildDimensionWrapperProperties(filterMaps);
+Map dimensionWrapperProperties = 
buildDimensionWrapperProperties();
 Map properties = 
buildWrapperProperties(ChildKeepingMergeWrapper.class.getName(),
 Collections.singletonList(
 buildWrapperProperties(DimensionWrapper.class.getName(), 
nestedPipelines, dimensionWrapperProperties)), this.mergerProperties);
 return new 
YamlTranslationResult().withProperties(properties).withComponents(this.components).withCron(cron);
   }
 
-  private Map buildDimensionWrapperProperties(Map> filterMaps) {
+  private Map buildDimensionWrapperProperties() {
 Map dimensionWrapperProperties = new HashMap<>();
 dimensionWrapperProperties.put(PROP_NESTED_METRIC_URNS, 
Collections.singletonList(this.metricUrn));
 if (yamlConfig.containsKey(PROP_DIMENSION_EXPLORATION)) {
@@ -470,7 +469,7 @@ public class CompositePipelineConfigTranslator extends 
YamlDetectionConfigTransl
 // Safety condition: Validate if maxDuration is greater than 15 minutes
 Map mergerProperties = MapUtils.getMap(yamlConfig, 
PROP_MERGER, new HashMap());
 if (mergerProperties.get(PROP_MAX_DURATION) != null) {
-  Preconditions.checkArgument(MapUtils.getLong(mergerProperties, 
PROP_MAX_DURATION) >= TimeUnit.MINUTES.toMillis(15),
+  Preconditions.checkArgument(MapUtils.getLong(mergerProperties, 
PROP_MAX_DURATION) >= datasetConfig.bucketTimeGranularity().toMillis(),
   "The maxDuration field set is not acceptable. Please check 

[GitHub] [incubator-pinot] jihaozh merged pull request #3982: [TE] yaml - more validation on max duration

[GitBox]

jihaozh merged pull request #3982: [TE] yaml - more validation on max duration
URL: https://github.com/apache/incubator-pinot/pull/3982
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jihaozh opened a new pull request #3982: [TE] yaml - more validation on max duration

[GitBox]

jihaozh opened a new pull request #3982: [TE] yaml - more validation on max 
duration
URL: https://github.com/apache/incubator-pinot/pull/3982
 
 
   - Validate that the max duration is larger than the data set granularity
   - Code clean up


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] jihaozh opened a new pull request #3981: [TE] detection - align metric slices

[GitBox]

jihaozh opened a new pull request #3981: [TE] detection - align metric slices
URL: https://github.com/apache/incubator-pinot/pull/3981
 
 
   Align metric slices to the data granularity.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: [TE] frontend - harleyjj/home - set default date picker to yesterday (#3976)

[jihao]

This is an automated email from the ASF dual-hosted git repository.

jihao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 636c6c1  [TE] frontend - harleyjj/home - set default date picker to 
yesterday (#3976)
636c6c1 is described below

commit 636c6c1c93a485cddc9f804ee471f22029f447d3
Author: Harley Jackson 
AuthorDate: Mon Mar 18 11:39:23 2019 -0700

[TE] frontend - harleyjj/home - set default date picker to yesterday (#3976)

Sets default date/time range on home page to be yesterday, to promote 
consistency of data from 0:00 to 0:00

Cleans up some eslint warnings
---
 thirdeye/thirdeye-frontend/app/pods/home/index/controller.js |  8 +++-
 thirdeye/thirdeye-frontend/app/pods/home/index/route.js  | 12 
 2 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/thirdeye/thirdeye-frontend/app/pods/home/index/controller.js 
b/thirdeye/thirdeye-frontend/app/pods/home/index/controller.js
index 256eea3..ec1543e 100644
--- a/thirdeye/thirdeye-frontend/app/pods/home/index/controller.js
+++ b/thirdeye/thirdeye-frontend/app/pods/home/index/controller.js
@@ -9,7 +9,7 @@ import * as anomalyUtil from 'thirdeye-frontend/utils/anomaly';
 import { inject as service } from '@ember/service';
 
 const TIME_PICKER_INCREMENT = 5; // tells date picker hours field how 
granularly to display time
-const DEFAULT_ACTIVE_DURATION = '1d'; // setting this date range selection as 
default (Last 24 Hours)
+const DEFAULT_ACTIVE_DURATION = 'today'; // setting this date range selection 
default as Today
 const UI_DATE_FORMAT = 'MMM D,  hh:mm a'; // format for date picker to use 
(usually varies by route or metric)
 const DISPLAY_DATE_FORMAT = '-MM-DD HH:mm'; // format used consistently 
across app to display custom date range
 const TIME_RANGE_OPTIONS = ['today', '1d', '2d', '1w'];
@@ -44,8 +44,6 @@ export default Controller.extend({
   sortedApplications: computed(
 'model.applications',
 function() {
-  let model = get(this, 'model');
-
   // Iterate through each anomaly
   let applications =  
this.get('store').peekAll('application').sortBy('application');
   return applications;
@@ -92,9 +90,9 @@ export default Controller.extend({
   const endDate = Number(get(this, 'model.endDate'));
   const duration = get(this, 'model.duration') || DEFAULT_ACTIVE_DURATION;
   const predefinedRanges = {
-'Today': [moment().startOf('day'), moment()],
+'Today': [moment().startOf('day'), moment().startOf('day').add(1, 
'day')],
 'Last 24 hours': [moment().subtract(1, 'day'), moment()],
-'Yesterday': [moment().subtract(1, 'day').startOf('day'), 
moment().subtract(1, 'days').endOf('day')],
+'Yesterday': [moment().subtract(1, 'day').startOf('day'), 
moment().startOf('day')],
 'Last Week': [moment().subtract(1, 'week'), moment()]
   };
 
diff --git a/thirdeye/thirdeye-frontend/app/pods/home/index/route.js 
b/thirdeye/thirdeye-frontend/app/pods/home/index/route.js
index a62b12e..86e70e2 100644
--- a/thirdeye/thirdeye-frontend/app/pods/home/index/route.js
+++ b/thirdeye/thirdeye-frontend/app/pods/home/index/route.js
@@ -1,14 +1,10 @@
 import Route from '@ember/routing/route';
 import columns from 'thirdeye-frontend/shared/anomaliesTableColumns';
-import fetch from 'fetch';
 import { hash } from 'rsvp';
-import { selfServeApiCommon } from 'thirdeye-frontend/utils/api/self-serve';
-import { setProperties } from '@ember/object';
 import { task } from 'ember-concurrency';
 import RSVP from "rsvp";
 import moment from 'moment';
 import { inject as service } from '@ember/service';
-import _ from 'lodash';
 import AuthenticatedRouteMixin from 
'ember-simple-auth/mixins/authenticated-route-mixin';
 
 const queryParamsConfig = {
@@ -29,9 +25,9 @@ export default Route.extend(AuthenticatedRouteMixin, {
   },
   applicationAnomalies: null,
   appName: null,
-  startDate: moment().subtract(1, 'day').utc().valueOf(), //taylored for Last 
24 hours vs Today -> moment().startOf('day').utc().valueOf(),
-  endDate: moment().utc().valueOf(),//taylored for Last 24 hours
-  duration: '1d',//taylored for Last 24 hours
+  startDate: moment().startOf('day').utc().valueOf(), //set default to 0:00 
for data consistency
+  endDate: moment().startOf('day').add(1, 'day').utc().valueOf(), //set 
default to 0:00 for data consistency
+  duration: 'today', //set default to today
   feedbackType: 'All Resolutions',
 
   /**
@@ -101,7 +97,7 @@ export default Route.extend(AuthenticatedRouteMixin, {
 });
   },
 
-  _getAnomalyMapping: task (function * (model) {//TODO: need to add to anomaly 
util - LH
+  _getAnomalyMapping: task (function * () {//TODO: need to add to anomaly util 
- LH
 let anomalyMapping = {};
 //fetch the anomalies from the onion wrapper cache.
 const applicationAnomalies = yield 

[GitHub] [incubator-pinot] jihaozh merged pull request #3976: [TE] frontend - harleyjj/home - set default date picker to yesterday

[GitBox]

jihaozh merged pull request #3976: [TE] frontend - harleyjj/home - set default 
date picker to yesterday
URL: https://github.com/apache/incubator-pinot/pull/3976
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] harleyjj opened a new pull request #3980: [TE] frontend - harleyjj/preview - default preview to 2 days to accom…

[GitBox]

harleyjj opened a new pull request #3980: [TE] frontend - harleyjj/preview - 
default preview to 2 days to accom…
URL: https://github.com/apache/incubator-pinot/pull/3980
 
 
   …odate daily metrics


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3979: Track Indexed timestamp across consuming segments

[GitBox]

mcvsubbu commented on a change in pull request #3979: Track Indexed timestamp 
across consuming segments
URL: https://github.com/apache/incubator-pinot/pull/3979#discussion_r266569347
 
 

 ##
 File path: 
pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
 ##
 @@ -204,20 +205,22 @@ public boolean index(GenericRow row) {
 // If metrics aggregation is enabled and if the dimension values were 
already seen, this will return existing docId,
 // else this will return a new docId.
 int docId = getOrCreateDocId(dictIdMap);
-
+boolean canTakeMore = false;
 // docId == numDocs implies new docId.
 if (docId == numDocs) {
   // Add forward and inverted indices for new document.
   addForwardIndex(row, docId, dictIdMap);
   addInvertedIndex(docId, dictIdMap);
   // Update number of document indexed at last to make the latest record 
queryable
-  return _numDocsIndexed++ < _capacity;
+  canTakeMore = _numDocsIndexed++ < _capacity;
 } else {
-  Preconditions
-  .checkState(_aggregateMetrics, "Invalid document-id during indexing: 
" + docId + " expected: " + numDocs);
+  Preconditions.checkState(_aggregateMetrics, "Invalid document-id during 
indexing: " + docId + " expected: " + numDocs);
   // Update metrics for existing document.
-  return aggregateMetrics(row, docId);
+  canTakeMore = aggregateMetrics(row, docId);
 }
+// update indexing time
+_lastIndexedTimestamp = System.currentTimeMillis();
 
 Review comment:
   Issues with indexing the row should be dealt with as a bug and fixed, since 
that is a loss of data. 
   
   Also, I was proposing to extend the interface so that a stream that keeps 
track of timestamps of incoming messages (for its own SLA) can use the 
mechanism. I was not referring to the time column of the schema
   
   We can chat a little to see if any of these ideas can be explored further 
before discarding


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.

[GitBox]

mcvsubbu commented on a change in pull request #3941: Pinot server side change 
to optimize LLC segment completion with direct metadata upload. 
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r266565308
 
 

 ##
 File path: 
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
 ##
 @@ -735,14 +742,45 @@ protected SegmentBuildDescriptor 
buildSegmentInternal(boolean forCommit) {
 if (_isOffHeap) {
   params.withMemoryUsedBytes(_memoryManager.getTotalAllocatedBytes());
 }
-SegmentCompletionProtocol.Response commitEndResponse = 
_protocolHandler.segmentCommitEnd(params);
+SegmentCompletionProtocol.Response commitEndResponse;
+if (!_indexLoadingConfig.isEnableSplitCommitEndWithMetadata()) {
+  commitEndResponse = _protocolHandler.segmentCommitEnd(params);
+} else {
+  Map metadataFiles = new HashMap<>();
+  try {
+metadataFiles.put(V1Constants.MetadataKeys.METADATA_FILE_NAME,
 
 Review comment:
   I think we keep the raw files and build the tar.gz in addition. The segment 
needs to be in raw format for serving anyways. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] chenboat commented on a change in pull request #3941: Pinot server side change to optimize LLC segment completion with direct metadata upload.

[GitBox]

chenboat commented on a change in pull request #3941: Pinot server side change 
to optimize LLC segment completion with direct metadata upload. 
URL: https://github.com/apache/incubator-pinot/pull/3941#discussion_r266561972
 
 

 ##
 File path: 
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
 ##
 @@ -735,14 +742,45 @@ protected SegmentBuildDescriptor 
buildSegmentInternal(boolean forCommit) {
 if (_isOffHeap) {
   params.withMemoryUsedBytes(_memoryManager.getTotalAllocatedBytes());
 }
-SegmentCompletionProtocol.Response commitEndResponse = 
_protocolHandler.segmentCommitEnd(params);
+SegmentCompletionProtocol.Response commitEndResponse;
+if (!_indexLoadingConfig.isEnableSplitCommitEndWithMetadata()) {
+  commitEndResponse = _protocolHandler.segmentCommitEnd(params);
+} else {
+  Map metadataFiles = new HashMap<>();
+  try {
+metadataFiles.put(V1Constants.MetadataKeys.METADATA_FILE_NAME,
 
 Review comment:
   Let me explore a bit your idea to pass in the metadata files via the Segment 
Descriptor.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on issue #3978: The segments are stored in memory

[GitBox]

mcvsubbu commented on issue #3978: The segments are stored in memory
URL: 
https://github.com/apache/incubator-pinot/issues/3978#issuecomment-473982286
 
 
   All segments of a table are divided among the tenant instances that host the 
table. If a server has a segment to serve, it will be loaded into memory.  You 
can use rest commands on the controller to see the idealstate of a table, and 
that will indicate to you the servers that host each segment. 
   
   It is up to you how you want to reduce cost.
   
   You can load the segments in "HEAP" mode, you just need to make sure that 
the maxDirectMemory argument to the jvm is sufficiently large so that the 
segments are loaded without OOM errors.
   
   You can load the segments in MMAP mode, in which case the segments are paged 
in from disk and Pinot lets the Operating System handle paging of segments in 
and out.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] sunithabeeram commented on a change in pull request #3979: Track Indexed timestamp across consuming segments

[GitBox]

sunithabeeram commented on a change in pull request #3979: Track Indexed 
timestamp across consuming segments
URL: https://github.com/apache/incubator-pinot/pull/3979#discussion_r266520116
 
 

 ##
 File path: 
pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
 ##
 @@ -204,20 +205,22 @@ public boolean index(GenericRow row) {
 // If metrics aggregation is enabled and if the dimension values were 
already seen, this will return existing docId,
 // else this will return a new docId.
 int docId = getOrCreateDocId(dictIdMap);
-
+boolean canTakeMore = false;
 // docId == numDocs implies new docId.
 if (docId == numDocs) {
   // Add forward and inverted indices for new document.
   addForwardIndex(row, docId, dictIdMap);
   addInvertedIndex(docId, dictIdMap);
   // Update number of document indexed at last to make the latest record 
queryable
-  return _numDocsIndexed++ < _capacity;
+  canTakeMore = _numDocsIndexed++ < _capacity;
 } else {
-  Preconditions
-  .checkState(_aggregateMetrics, "Invalid document-id during indexing: 
" + docId + " expected: " + numDocs);
+  Preconditions.checkState(_aggregateMetrics, "Invalid document-id during 
indexing: " + docId + " expected: " + numDocs);
   // Update metrics for existing document.
-  return aggregateMetrics(row, docId);
+  canTakeMore = aggregateMetrics(row, docId);
 }
+// update indexing time
+_lastIndexedTimestamp = System.currentTimeMillis();
 
 Review comment:
   The user defined SLA will account for any expected delays in the incoming 
stream itself. Just going by the poll time can miss cases where we have issues 
with indexing (there have been a few in the past; for ex, issues with 
aggregatable-metrics for multi-value columns etc).
   
   The timestamp in the record can have a very coarse granularity, for ex, days 
- which isn't ideal for the granularity we would like to track freshness at. 
Also, see motivation notes about the lack of other usable timestamps.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3979: Track Indexed timestamp across consuming segments

[GitBox]

mcvsubbu commented on a change in pull request #3979: Track Indexed timestamp 
across consuming segments
URL: https://github.com/apache/incubator-pinot/pull/3979#discussion_r266511438
 
 

 ##
 File path: 
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
 ##
 @@ -168,6 +180,14 @@ public BrokerResponseNative reduceOnDataTable(@Nonnull 
BrokerRequest brokerReque
 brokerResponseNative.setNumSegmentsMatched(numSegmentsMatched);
 brokerResponseNative.setTotalDocs(numTotalRawDocs);
 brokerResponseNative.setNumGroupsLimitReached(numGroupsLimitReached);
+if (numConsumingQueried > 0) {
+  if (minConsumingIndexTs == Long.MAX_VALUE) {
+LOGGER.error("Invalid lastIndexedTimestamp across {} consuming 
segments", numConsumingQueried);
 
 Review comment:
   Log the query ID here so that we can use it to correlate query logs. Also, 
please ensure that if this condition happens we override the throttle in the 
broker logs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3979: Track Indexed timestamp across consuming segments

[GitBox]

mcvsubbu commented on a change in pull request #3979: Track Indexed timestamp 
across consuming segments
URL: https://github.com/apache/incubator-pinot/pull/3979#discussion_r266514769
 
 

 ##
 File path: 
pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
 ##
 @@ -204,20 +205,22 @@ public boolean index(GenericRow row) {
 // If metrics aggregation is enabled and if the dimension values were 
already seen, this will return existing docId,
 // else this will return a new docId.
 int docId = getOrCreateDocId(dictIdMap);
-
+boolean canTakeMore = false;
 // docId == numDocs implies new docId.
 if (docId == numDocs) {
   // Add forward and inverted indices for new document.
   addForwardIndex(row, docId, dictIdMap);
   addInvertedIndex(docId, dictIdMap);
   // Update number of document indexed at last to make the latest record 
queryable
-  return _numDocsIndexed++ < _capacity;
+  canTakeMore = _numDocsIndexed++ < _capacity;
 } else {
-  Preconditions
-  .checkState(_aggregateMetrics, "Invalid document-id during indexing: 
" + docId + " expected: " + numDocs);
+  Preconditions.checkState(_aggregateMetrics, "Invalid document-id during 
indexing: " + docId + " expected: " + numDocs);
   // Update metrics for existing document.
-  return aggregateMetrics(row, docId);
+  canTakeMore = aggregateMetrics(row, docId);
 }
+// update indexing time
+_lastIndexedTimestamp = System.currentTimeMillis();
 
 Review comment:
   If there is a delay in the stream, that will show up as a pinot delay. 
Another choice is to take the last _polled_ timestamp. That will ensure that as 
long as Pinot polls the stream, the available data is considered fresh. Not 
sure why we are not doing that.  A stream might (and probably should) have its 
own staleness metric being tracked and monitored so that an application does 
not need to track a staleness metric via pinot and root cause it to the stream 
if that is indeed the case.
   
   If you want to continue with this approach, then I suggest that you extend 
the interface to retrieve rows from a stream to include the highest available 
timestamp of a record _in_ the stream. That way, streams that provide the 
staleness metric can translate (via Pinot) to the staleness monitoring 
application.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #3979: Track Indexed timestamp across consuming segments

[GitBox]

mcvsubbu commented on a change in pull request #3979: Track Indexed timestamp 
across consuming segments
URL: https://github.com/apache/incubator-pinot/pull/3979#discussion_r266506853
 
 

 ##
 File path: 
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
 ##
 @@ -203,6 +205,27 @@ public void setNumSegmentsMatched(long 
numSegmentsMatched) {
 _numSegmentsMatched = numSegmentsMatched;
   }
 
+  @JsonProperty("numConsumingQueried")
+  public long getNumConsumingQueried() {
+return _numConsumingQueried;
+  }
+
+  @JsonProperty("numConsumingQueried")
+  public void setNumConsumingQueried(long queried) {
+_numConsumingQueried = queried;
+  }
+
+  @JsonProperty("minConsumingIndexTs")
 
 Review comment:
   Instead of 'Ts' could we suffix it with Millis (or Seconds, whatever the 
units may be)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] codecov-io commented on issue #3979: Track Indexed timestamp across consuming segments

[GitBox]

codecov-io commented on issue #3979: Track Indexed timestamp across consuming 
segments
URL: https://github.com/apache/incubator-pinot/pull/3979#issuecomment-473929895
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3979?src=pr=h1) 
Report
   > Merging 
[#3979](https://codecov.io/gh/apache/incubator-pinot/pull/3979?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-pinot/commit/b2c5a882fb35f1be7bcaeb0f6c65c28a8f1d7ff9?src=pr=desc)
 will **increase** coverage by `10.04%`.
   > The diff coverage is `88.23%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-pinot/pull/3979/graphs/tree.svg?width=650=4ibza2ugkz=150=pr)](https://codecov.io/gh/apache/incubator-pinot/pull/3979?src=pr=tree)
   
   ```diff
   @@  Coverage Diff  @@
   ## master#3979   +/-   ##
   =
   + Coverage 57.01%   67.05%   +10.04% 
 Complexity44   
   =
 Files  1032 1033+1 
 Lines 5105151098   +47 
 Branches   7126 7136   +10 
   =
   + Hits  2910634264 +5158 
   + Misses1979014489 -5301 
   - Partials   2155 2345  +190
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-pinot/pull/3979?src=pr=tree) | 
Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[.../core/indexsegment/mutable/MutableSegmentImpl.java](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pbmRleHNlZ21lbnQvbXV0YWJsZS9NdXRhYmxlU2VnbWVudEltcGwuamF2YQ==)
 | `67.19% <100%> (+20.27%)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...g/apache/pinot/core/indexsegment/IndexSegment.java](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pbmRleHNlZ21lbnQvSW5kZXhTZWdtZW50LmphdmE=)
 | `100% <100%> (ø)` | `0 <0> (?)` | |
   | 
[...e/indexsegment/immutable/ImmutableSegmentImpl.java](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRJbXBsLmphdmE=)
 | `69.35% <100%> (+0.5%)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...t/common/response/broker/BrokerResponseNative.java](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVzcG9uc2UvYnJva2VyL0Jyb2tlclJlc3BvbnNlTmF0aXZlLmphdmE=)
 | `91.3% <100%> (+5.59%)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...e/pinot/core/query/reduce/BrokerReduceService.java](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvQnJva2VyUmVkdWNlU2VydmljZS5qYXZh)
 | `79.38% <78.57%> (+23.74%)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...core/query/executor/ServerQueryExecutorV1Impl.java](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9leGVjdXRvci9TZXJ2ZXJRdWVyeUV4ZWN1dG9yVjFJbXBsLmphdmE=)
 | `79.36% <82.35%> (+21.56%)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...n/java/org/apache/pinot/common/data/FieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZGF0YS9GaWVsZFNwZWMuamF2YQ==)
 | `73.88% <0%> (-0.64%)` | `0% <0%> (ø)` | |
   | 
[...ator/transform/function/BaseTransformFunction.java](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vQmFzZVRyYW5zZm9ybUZ1bmN0aW9uLmphdmE=)
 | `29.95% <0%> (+0.42%)` | `0% <0%> (ø)` | :arrow_down: |
   | 
[...g/apache/pinot/common/utils/helix/HelixHelper.java](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaGVsaXgvSGVsaXhIZWxwZXIuamF2YQ==)
 | `56.25% <0%> (+0.56%)` | `0% <0%> (ø)` | :arrow_down: |
   | ... and [335 
more](https://codecov.io/gh/apache/incubator-pinot/pull/3979/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3979?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/3979?src=pr=footer).
 Last update 

[GitHub] [incubator-pinot] sunithabeeram merged pull request #3977: Minor spelling correction

[GitBox]

sunithabeeram merged pull request #3977: Minor spelling correction
URL: https://github.com/apache/incubator-pinot/pull/3977
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: Spelling correction (#3977)

[sunithabeeram]

This is an automated email from the ASF dual-hosted git repository.

sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 581464e  Spelling correction (#3977)
581464e is described below

commit 581464ee63529b53ba528f84e970f50e0e2e7d5e
Author: Shrinivas Devshatwar <47772040+sdevshat...@users.noreply.github.com>
AuthorDate: Mon Mar 18 06:47:53 2019 -0700

Spelling correction (#3977)
---
 docs/client_api.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/client_api.rst b/docs/client_api.rst
index e9c1a16..04e9bf0 100644
--- a/docs/client_api.rst
+++ b/docs/client_api.rst
@@ -20,7 +20,7 @@
 Executing queries via REST API on the Broker
 
 
-The Pinot REST API can be accessed by invoking ``POST`` operation witha a JSON 
body containing the parameter ``pql``
+The Pinot REST API can be accessed by invoking ``POST`` operation with a JSON 
body containing the parameter ``pql``
 to the ``/query`` URI endpoint on a broker. Depending on the type of query, 
the results can take different shapes.
 The examples below use curl.
 


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] sunithabeeram commented on issue #3977: Minor spelling correction

[GitBox]

sunithabeeram commented on issue #3977: Minor spelling correction
URL: https://github.com/apache/incubator-pinot/pull/3977#issuecomment-473916583
 
 
   FlakyConsumerRealtimeClusterIntegrationTest seems to have failed. Created a 
tracker for it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] sunithabeeram opened a new pull request #3979: Track Indexed timestamp across consuming segments

[GitBox]

sunithabeeram opened a new pull request #3979: Track Indexed timestamp across 
consuming segments
URL: https://github.com/apache/incubator-pinot/pull/3979
 
 
   *Motivation*: We would like to track freshness of the realtime data being 
consumed and available for queries. The true indication of this could be a 
timestamp for when the message was accepted into the stream-provider's (such as 
Kafka's) queue. However, this isn't something that is exposed as a client API. 
This change exposes the lastIndexed timestamp of a consuming segment as a proxy 
for freshness.
   Exposing this through the query path helps comparison against client defined 
SLAs and provides ability to reflect freshness as part of Pinot's availability.
   
   *Changes*: The PR exposes lastIndexedTimestamp on the MutableSegment 
interface. The min timestamp across consuming segments is obtained by the 
server query-executor before the segments are pruned out. This along with the 
number of consuming segments queried is returned to the broker in the response.
   
   *Testing done*: Unit and integration tests added.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] ly923976094 opened a new issue #3978: The segments are stored in memory

[GitBox]

ly923976094 opened a new issue #3978: The segments are stored in memory
URL: https://github.com/apache/incubator-pinot/issues/3978
 
 
   Once segments are assigned, Pinot servers get notified via Helix to “host” 
the segment. The servers download the segments (as a cached local copy to serve 
queries) and load them into local memory. All segment data is maintained in 
memory as long as the server hosts that segment.
   
   I want to know which segments are permanently loaded into the memory, and 
whether the memory is expensive and the data volume is large


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] ly923976094 closed issue #3963: Tools for pinot data migration (real time and offline)

[GitBox]

ly923976094 closed issue #3963: Tools for pinot data migration (real time and 
offline)
URL: https://github.com/apache/incubator-pinot/issues/3963
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org