[incubator-pinot] branch master updated (f015c68 -> 1b8cdd8)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from f015c68 Add sql query command (#4726) add 1b8cdd8 Fix comparison to pick the right minimum value for hash-set (#4588) No new revisions were added by this update. Summary of changes: pinot-common/src/main/java/org/apache/pinot/common/utils/HashUtil.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Expose table-name via StreamConfig (#4543)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new d261a76 Expose table-name via StreamConfig (#4543)
d261a76 is described below
commit d261a76e3b78eeec0d560d60c8f1999df5d7b7d0
Author: Sunitha Beeram
AuthorDate: Tue Aug 20 08:22:09 2019 -0700
Expose table-name via StreamConfig (#4543)
---
.../broker/routing/RoutingTableBuilderFactory.java | 3 +-
.../impl/kafka/KafkaLowLevelStreamConfigTest.java | 2 +-
.../kafka/KafkaPartitionLevelConsumerTest.java | 14 +++--
.../kafka2/KafkaPartitionLevelConsumerTest.java| 19 +++---
.../KafkaPartitionLevelStreamConfigTest.java | 3 +-
.../helix/core/PinotHelixResourceManager.java | 4 +-
.../helix/core/PinotTableIdealStateBuilder.java| 2 +-
.../realtime/PinotLLCRealtimeSegmentManager.java | 11 ++--
.../core/realtime/PinotRealtimeSegmentManager.java | 5 +-
.../segment/FlushThresholdUpdateManager.java | 2 +-
.../rebalance/DefaultRebalanceSegmentStrategy.java | 6 +-
.../RealtimeSegmentValidationManager.java | 4 +-
.../realtime/HLRealtimeSegmentDataManager.java | 2 +-
.../realtime/LLRealtimeSegmentDataManager.java | 2 +-
.../stream/PartitionLevelStreamConfig.java | 5 +-
.../pinot/core/realtime/stream/StreamConfig.java | 17 +-
.../apache/pinot/core/util/ReplicationUtils.java | 7 ++-
.../impl/fakestream/FakeStreamConfigUtils.java | 5 +-
.../core/realtime/stream/StreamConfigTest.java | 67 --
19 files changed, 108 insertions(+), 72 deletions(-)
diff --git
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingTableBuilderFactory.java
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingTableBuilderFactory.java
index 574ccf5..a9b897f 100644
---
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingTableBuilderFactory.java
+++
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingTableBuilderFactory.java
@@ -131,7 +131,8 @@ public class RoutingTableBuilderFactory {
break;
case PartitionAwareRealtime:
// Check that the table uses LL consumer.
-StreamConfig streamConfig = new
StreamConfig(tableConfig.getIndexingConfig().getStreamConfigs());
+StreamConfig streamConfig = new
StreamConfig(tableConfig.getTableName(),
+tableConfig.getIndexingConfig().getStreamConfigs());
if (streamConfig.getConsumerTypes().size() == 1 &&
streamConfig.hasLowLevelConsumerType()) {
builder = new PartitionAwareRealtimeRoutingTableBuilder();
diff --git
a/pinot-connectors/pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaLowLevelStreamConfigTest.java
b/pinot-connectors/pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaLowLevelStreamConfigTest.java
index bdb48cb..d224d4d 100644
---
a/pinot-connectors/pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaLowLevelStreamConfigTest.java
+++
b/pinot-connectors/pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaLowLevelStreamConfigTest.java
@@ -66,7 +66,7 @@ public class KafkaLowLevelStreamConfigTest {
if (fetcherMinBytes != null) {
streamConfigMap.put("stream.kafka.fetcher.minBytes", fetcherMinBytes);
}
-return new KafkaLowLevelStreamConfig(new StreamConfig(streamConfigMap));
+return new KafkaLowLevelStreamConfig(new
StreamConfig("fakeTable_REALTIME", streamConfigMap));
}
@Test
diff --git
a/pinot-connectors/pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaPartitionLevelConsumerTest.java
b/pinot-connectors/pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaPartitionLevelConsumerTest.java
index 64788b7..9b6b950 100644
---
a/pinot-connectors/pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaPartitionLevelConsumerTest.java
+++
b/pinot-connectors/pinot-connector-kafka-0.9/src/test/java/org/apache/pinot/core/realtime/impl/kafka/KafkaPartitionLevelConsumerTest.java
@@ -222,6 +222,7 @@ public class KafkaPartitionLevelConsumerTest {
String streamKafkaBrokerList = "abcd:1234,bcde:2345";
String streamKafkaConsumerType = "simple";
String clientId = "clientId";
+String tableNameWithType = "table_REALTIME";
MockKafkaSimpleConsumerFactory mockKafkaSimpleConsumerFactory =
new MockKafkaSimpleConsumerFactory(new String[]{"abcd", "bcde"}, new
int[]{1234, 2345},
@@ -237,7 +238,7 @@ public class KafkaPartitionLevelConsumerTest {
streamConfigMap.put("stream.kafka.decod
[incubator-pinot] branch master updated: Expose a method to determine if a QueryExceptionErrorCode represents a client-side error (#4532)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 83cc88d Expose a method to determine if a QueryExceptionErrorCode
represents a client-side error (#4532)
83cc88d is described below
commit 83cc88d76298edafbd0ca72a7d2fbde8e13dabf0
Author: Sunitha Beeram
AuthorDate: Mon Aug 19 21:21:25 2019 -0700
Expose a method to determine if a QueryExceptionErrorCode represents a
client-side error (#4532)
* Expose a method to determine if a QueryExceptionErrorCode represents a
client-side error
* Address review comments
---
.../apache/pinot/broker/api/RequestStatistics.java | 9 +++-
.../pinot/common/exception/QueryException.java | 24 ++
2 files changed, 32 insertions(+), 1 deletion(-)
diff --git
a/pinot-broker/src/main/java/org/apache/pinot/broker/api/RequestStatistics.java
b/pinot-broker/src/main/java/org/apache/pinot/broker/api/RequestStatistics.java
index 5607cb0..cb82312 100644
---
a/pinot-broker/src/main/java/org/apache/pinot/broker/api/RequestStatistics.java
+++
b/pinot-broker/src/main/java/org/apache/pinot/broker/api/RequestStatistics.java
@@ -28,9 +28,12 @@ import org.apache.pinot.common.response.BrokerResponse;
* post-processing at a finer level than metrics.
*/
public class RequestStatistics {
+
+ private static final String DEFAULT_TABLE_NAME = "NotYetParsed";
+
private int _errorCode = 0;
private String _pql;
- private String _tableName = "NotYetParsed";
+ private String _tableName = DEFAULT_TABLE_NAME;
private long _processingTimeMillis = -1;
private long _totalDocs;
@@ -189,4 +192,8 @@ public class RequestStatistics {
public int getNumExceptions() {
return _numExceptions;
}
+
+ public boolean hasValidTableName() {
+return ! DEFAULT_TABLE_NAME.equals(_tableName);
+ }
}
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
b/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
index 6c45897..6f83c8d 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
@@ -61,6 +61,7 @@ public class QueryException {
public static final int COMBINE_GROUP_BY_EXCEPTION_ERROR_CODE = 600;
public static final int QUERY_VALIDATION_ERROR_CODE = 700;
public static final int UNKNOWN_ERROR_CODE = 1000;
+ // NOTE: update isClientError() method appropriately when new codes are added
public static final ProcessingException JSON_PARSING_ERROR = new
ProcessingException(JSON_PARSING_ERROR_CODE);
public static final ProcessingException JSON_COMPILATION_ERROR = new
ProcessingException(JSON_COMPILATION_ERROR_CODE);
@@ -147,4 +148,27 @@ public class QueryException {
copiedProcessingException.setMessage(errorType + ":\n" + errorMessage);
return copiedProcessingException;
}
+
+ /**
+ * Determines if a query-exception-error-code represents an error on the
client side.
+ * @param errorCode the error code from processing the query
+ * @return whether the code indicates client error or not
+ */
+ public static boolean isClientError(int errorCode) {
+switch (errorCode) {
+ // NOTE: QueryException.BROKER_RESOURCE_MISSING_ERROR can be triggered
either due to
+ // client error (incorrect table name) or due to issues with EV updates.
For cases where
+ // access to tables is controlled via ACLs, for an incorrect table name
we expect ACCESS_DENIED_ERROR to be
+ // thrown. Hence, we currently don't treat BROKER_RESOURCE_MISSING_ERROR
as client error.
+ case QueryException.ACCESS_DENIED_ERROR_CODE:
+ case QueryException.JSON_COMPILATION_ERROR_CODE:
+ case QueryException.JSON_PARSING_ERROR_CODE:
+ case QueryException.QUERY_VALIDATION_ERROR_CODE:
+ case QueryException.PQL_PARSING_ERROR_CODE:
+ case QueryException.TOO_MANY_REQUESTS_ERROR_CODE:
+return true;
+ default:
+return false;
+}
+ }
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Rename metric to reflect what's being counted: segments-processed vs segments-queried (#4499)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new ac1f387 Rename metric to reflect what's being counted:
segments-processed vs segments-queried (#4499)
ac1f387 is described below
commit ac1f38702d861852a0b5838a614df87673bf7f9a
Author: Sunitha Beeram
AuthorDate: Tue Aug 6 11:18:29 2019 -0700
Rename metric to reflect what's being counted: segments-processed vs
segments-queried (#4499)
---
.../main/java/org/apache/pinot/common/utils/DataTable.java | 2 +-
.../pinot/core/query/executor/ServerQueryExecutorV1Impl.java | 12 ++--
.../apache/pinot/core/query/reduce/BrokerReduceService.java | 12 ++--
.../apache/pinot/core/query/scheduler/QueryScheduler.java| 2 +-
4 files changed, 14 insertions(+), 14 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java
index 5383a3c..780d57f 100644
--- a/pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java
+++ b/pinot-common/src/main/java/org/apache/pinot/common/utils/DataTable.java
@@ -35,7 +35,7 @@ public interface DataTable {
String NUM_SEGMENTS_QUERIED = "numSegmentsQueried";
String NUM_SEGMENTS_PROCESSED = "numSegmentsProcessed";
String NUM_SEGMENTS_MATCHED = "numSegmentsMatched";
- String NUM_CONSUMING_SEGMENTS_QUERIED = "numConsumingSegmentsQueried";
+ String NUM_CONSUMING_SEGMENTS_PROCESSED = "numConsumingSegmentsProcessed";
String MIN_CONSUMING_FRESHNESS_TIME_MS = "minConsumingFreshnessTimeMs";
String TOTAL_DOCS_METADATA_KEY = "totalDocs";
String NUM_GROUPS_LIMIT_REACHED_KEY = "numGroupsLimitReached";
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java
b/pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java
index 6b59e1a..b60471e 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/query/executor/ServerQueryExecutorV1Impl.java
@@ -151,13 +151,13 @@ public class ServerQueryExecutorV1Impl implements
QueryExecutor {
TraceContext.register(requestId);
}
-int numConsumingSegmentsQueried = 0;
+int numConsumingSegmentsProcessed = 0;
long minIndexTimeMs = Long.MAX_VALUE;
long minIngestionTimeMs = Long.MAX_VALUE;
// gather stats for realtime consuming segments
for (SegmentDataManager segmentMgr : segmentDataManagers) {
if (segmentMgr.getSegment() instanceof MutableSegment) {
-numConsumingSegmentsQueried += 1;
+numConsumingSegmentsProcessed += 1;
SegmentMetadata metadata =
segmentMgr.getSegment().getSegmentMetadata();
long indexedTime = metadata.getLastIndexedTimestamp();
if (indexedTime != Long.MIN_VALUE && indexedTime < minIndexTimeMs) {
@@ -171,12 +171,12 @@ public class ServerQueryExecutorV1Impl implements
QueryExecutor {
}
long minConsumingFreshnessTimeMs = minIngestionTimeMs;
-if (numConsumingSegmentsQueried > 0) {
+if (numConsumingSegmentsProcessed > 0) {
if (minIngestionTimeMs == Long.MAX_VALUE) {
LOGGER.debug("Did not find valid ingestionTimestamp across consuming
segments! Using indexTime instead");
minConsumingFreshnessTimeMs = minIndexTimeMs;
}
- LOGGER.debug("Querying {} consuming segments with min
minConsumingFreshnessTimeMs {}", numConsumingSegmentsQueried,
minConsumingFreshnessTimeMs);
+ LOGGER.debug("Querying {} consuming segments with min
minConsumingFreshnessTimeMs {}", numConsumingSegmentsProcessed,
minConsumingFreshnessTimeMs);
}
DataTable dataTable = null;
@@ -256,8 +256,8 @@ public class ServerQueryExecutorV1Impl implements
QueryExecutor {
_serverMetrics.addMeteredTableValue(tableNameWithType,
ServerMeter.NUM_MISSING_SEGMENTS, missingSegments);
}
-if (numConsumingSegmentsQueried > 0) {
- dataTable.getMetadata().put(DataTable.NUM_CONSUMING_SEGMENTS_QUERIED,
Integer.toString(numConsumingSegmentsQueried));
+if (numConsumingSegmentsProcessed > 0) {
+ dataTable.getMetadata().put(DataTable.NUM_CONSUMING_SEGMENTS_PROCESSED,
Integer.toString(numConsumingSegmentsProcessed));
dataTable.getMetadata().put(DataTable.MIN_CONSUMING_FRESHNESS_TIME_MS,
Long.toString(minConsumingFreshnessTimeMs));
}
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
b/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
index ca9d91f..acfc0fc 100644
-
[incubator-pinot] branch master updated: Emit freshness lag at server level as well (#4308)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new ad6a46c Emit freshness lag at server level as well (#4308)
ad6a46c is described below
commit ad6a46c3e472022ed23764b7f91aa19db97a1092
Author: Sunitha Beeram
AuthorDate: Mon Jun 17 10:34:09 2019 -0700
Emit freshness lag at server level as well (#4308)
* Emit freshness lag at server level as well
* Address review comments
---
.../java/org/apache/pinot/common/metadata/RowMetadata.java| 11 ---
.../java/org/apache/pinot/common/metrics/ServerTimer.java | 6 +-
.../pinot/core/indexsegment/mutable/MutableSegmentImpl.java | 3 +--
.../org/apache/pinot/core/query/scheduler/QueryScheduler.java | 6 ++
.../pinot/core/realtime/stream/StreamMessageMetadata.java | 6 ++
5 files changed, 26 insertions(+), 6 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metadata/RowMetadata.java
b/pinot-common/src/main/java/org/apache/pinot/common/metadata/RowMetadata.java
index 4e67a05..63be9f1 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metadata/RowMetadata.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metadata/RowMetadata.java
@@ -23,16 +23,21 @@ import org.apache.pinot.annotations.InterfaceStability;
/**
- * A class that provides relevant row-level metadata for rows ingested into a
segment.
+ * A class that provides relevant row-level metadata for rows indexed into a
segment.
*
- * Currently this is relevant for rows ingested into a mutable segment.
+ * Currently this is relevant for rows ingested into a mutable segment - the
metadata is expected to be
+ * provided by the underlying stream.
*/
@InterfaceAudience.Public
@InterfaceStability.Evolving
public interface RowMetadata {
/**
- * Return the ingestion timestamp of the row.
+ * Return the timestamp associated with when the row was ingested upstream.
+ * Expected to be mainly used for stream-based sources.
+ *
+ * @return timestamp (epoch in milliseconds) when the row was ingested
upstream
+ * Long.MIN_VALUE if not available
*/
long getIngestionTimeMs();
}
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java
index a8560f0..326ae10 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ServerTimer.java
@@ -26,7 +26,11 @@ import org.apache.pinot.common.Utils;
*
*/
public enum ServerTimer implements AbstractMetrics.Timer {
- CURRENT_MSG_EVENT_TIMESTAMP_LAG("currentMsgEventTimestampLag", false);
+ // don't see usages for this
+ @Deprecated
+ CURRENT_MSG_EVENT_TIMESTAMP_LAG("currentMsgEventTimestampLag", false),
+ // metric tracking the freshness lag for consuming segments
+ FRESHNESS_LAG_MS("freshnessLagMs", false);
private final String timerName;
private final boolean global;
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
index 3a09bbb..972d91c 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
@@ -99,7 +99,6 @@ public class MutableSegmentImpl implements MutableSegment {
private final int _numKeyColumns;
// default message metadata
- private static final StreamMessageMetadata _defaultMetadata = new
StreamMessageMetadata(System.currentTimeMillis());
private volatile long _lastIndexedTimeMs = Long.MIN_VALUE;
private volatile long _latestIngestionTimeMs = Long.MIN_VALUE;
@@ -243,7 +242,7 @@ public class MutableSegmentImpl implements MutableSegment {
_lastIndexedTimeMs = System.currentTimeMillis();
-if (rowMetadata != null) {
+if (rowMetadata != null && rowMetadata.getIngestionTimeMs() !=
Long.MIN_VALUE) {
_latestIngestionTimeMs = Math.max(_latestIngestionTimeMs,
rowMetadata.getIngestionTimeMs());
}
return canTakeMore;
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java
b/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java
index 51e28ed..24c4ec0 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java
@@ -25,6 +25,7 @@ import com.google.common.util.concurrent.ListenableFutureTask;
import com.google.commo
[incubator-pinot] branch master updated: skip bit-map inverted index generation for sorted column (#4270)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 9f6c20b skip bit-map inverted index generation for sorted column
(#4270)
9f6c20b is described below
commit 9f6c20bfe07b4356a5e5fec5deb3f8337eed3c16
Author: JasperJiaGuo
AuthorDate: Mon Jun 10 05:36:48 2019 -0700
skip bit-map inverted index generation for sorted column (#4270)
* skip the inverted index generation
* skip the cleaning, no potential side effect
* fixed comments
* fixed comments
* fixed comments
* standalone test done
---
.../pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
index 14ba041..2016776 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
@@ -157,8 +157,8 @@ public class SegmentColumnarIndexCreator implements
SegmentCreator {
indexCreationInfo.getTotalNumberOfEntries()));
}
-// Initialize inverted index creator
-if (invertedIndexColumns.contains(columnName)) {
+// Initialize inverted index creator; skip creating inverted index if
sorted
+if (invertedIndexColumns.contains(columnName) &&
!indexCreationInfo.isSorted()) {
if (segmentCreationSpec.isOnHeap()) {
_invertedIndexCreatorMap
.put(columnName, new
OnHeapBitmapInvertedIndexCreator(_indexDir, columnName, cardinality));
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch createSegments updated (ee551e2 -> 27545ba)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch createSegments in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard ee551e2 Add support for custom record-readers in the create-segment tool add 27545ba Add support for custom record-readers in the create-segment tool This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (ee551e2) \ N -- N -- N refs/heads/createSegments (27545ba) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: pinot-integration-tests/pom.xml | 3 +++ pinot-tools/pom.xml | 7 +-- 2 files changed, 8 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch createSegments updated (bdb2648 -> ee551e2)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch createSegments in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard bdb2648 Add support for custom record-readers in the create-segment tool new ee551e2 Add support for custom record-readers in the create-segment tool This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (bdb2648) \ N -- N -- N refs/heads/createSegments (ee551e2) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Add support for custom record-readers in the create-segment tool
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch createSegments
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit ee551e23043bc43bd13153bc8c00ccc691047ea1
Author: Sunitha Beeram
AuthorDate: Tue May 28 10:38:18 2019 -0700
Add support for custom record-readers in the create-segment tool
---
pinot-integration-tests/pom.xml| 6 ++
pinot-tools/pom.xml| 4
.../tools/admin/command/CreateSegmentCommand.java | 25 +-
3 files changed, 25 insertions(+), 10 deletions(-)
diff --git a/pinot-integration-tests/pom.xml b/pinot-integration-tests/pom.xml
index 20ad7c8..01c0532 100644
--- a/pinot-integration-tests/pom.xml
+++ b/pinot-integration-tests/pom.xml
@@ -126,6 +126,12 @@
org.apache.pinot
pinot-tools
+
+
+ org.apache.pinot
+ pinot-orc
+
+
com.101tec
diff --git a/pinot-tools/pom.xml b/pinot-tools/pom.xml
index 766852c..9fcd62e 100644
--- a/pinot-tools/pom.xml
+++ b/pinot-tools/pom.xml
@@ -44,6 +44,10 @@
org.apache.pinot
+ pinot-orc
+
+
+ org.apache.pinot
pinot-server
diff --git
a/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/CreateSegmentCommand.java
b/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/CreateSegmentCommand.java
index 95c1371..af22b87 100644
---
a/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/CreateSegmentCommand.java
+++
b/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/CreateSegmentCommand.java
@@ -228,15 +228,17 @@ public class CreateSegmentCommand extends
AbstractBaseAdminCommand implements Co
}
}
-FileFormat configFormat = segmentGeneratorConfig.getFormat();
-if (_format == null) {
- if (configFormat == null) {
-throw new RuntimeException("Format cannot be null in config file.");
- }
- _format = configFormat;
-} else {
- if (configFormat != _format && configFormat != FileFormat.AVRO) {
-LOGGER.warn("Find format conflict in command line and config file, use
config in command line: {}", _format);
+if (segmentGeneratorConfig.getRecordReaderPath() == null) {
+ FileFormat configFormat = segmentGeneratorConfig.getFormat();
+ if (_format == null) {
+if (configFormat == null) {
+ throw new RuntimeException("Either recordReaderPath (via
generatorConfigFile option) or format must be specified.");
+}
+_format = configFormat;
+ } else {
+if (configFormat != _format && configFormat != FileFormat.AVRO) {
+ LOGGER.warn("Find format conflict in command line and config file,
use config in command line: {}", _format);
+}
}
}
@@ -291,7 +293,10 @@ public class CreateSegmentCommand extends
AbstractBaseAdminCommand implements Co
File[] files = dir.listFiles(new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
-return name.toLowerCase().endsWith(_format.toString().toLowerCase());
+if (_format != null) {
+ return name.toLowerCase().endsWith(_format.toString().toLowerCase());
+}
+return true;
}
});
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Add support for custom record-readers in the create-segment tool
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch createSegments
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit bdb264861c502b992c764c80ef6701efec0225b4
Author: Sunitha Beeram
AuthorDate: Tue May 28 10:38:18 2019 -0700
Add support for custom record-readers in the create-segment tool
---
pinot-integration-tests/pom.xml| 6 ++
pinot-tools/pom.xml| 4
.../tools/admin/command/CreateSegmentCommand.java | 25 +-
3 files changed, 25 insertions(+), 10 deletions(-)
diff --git a/pinot-integration-tests/pom.xml b/pinot-integration-tests/pom.xml
index 20ad7c8..01c0532 100644
--- a/pinot-integration-tests/pom.xml
+++ b/pinot-integration-tests/pom.xml
@@ -126,6 +126,12 @@
org.apache.pinot
pinot-tools
+
+
+ org.apache.pinot
+ pinot-orc
+
+
com.101tec
diff --git a/pinot-tools/pom.xml b/pinot-tools/pom.xml
index 766852c..9fcd62e 100644
--- a/pinot-tools/pom.xml
+++ b/pinot-tools/pom.xml
@@ -44,6 +44,10 @@
org.apache.pinot
+ pinot-orc
+
+
+ org.apache.pinot
pinot-server
diff --git
a/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/CreateSegmentCommand.java
b/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/CreateSegmentCommand.java
index 95c1371..af22b87 100644
---
a/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/CreateSegmentCommand.java
+++
b/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/CreateSegmentCommand.java
@@ -228,15 +228,17 @@ public class CreateSegmentCommand extends
AbstractBaseAdminCommand implements Co
}
}
-FileFormat configFormat = segmentGeneratorConfig.getFormat();
-if (_format == null) {
- if (configFormat == null) {
-throw new RuntimeException("Format cannot be null in config file.");
- }
- _format = configFormat;
-} else {
- if (configFormat != _format && configFormat != FileFormat.AVRO) {
-LOGGER.warn("Find format conflict in command line and config file, use
config in command line: {}", _format);
+if (segmentGeneratorConfig.getRecordReaderPath() == null) {
+ FileFormat configFormat = segmentGeneratorConfig.getFormat();
+ if (_format == null) {
+if (configFormat == null) {
+ throw new RuntimeException("Either recordReaderPath (via
generatorConfigFile option) or format must be specified.");
+}
+_format = configFormat;
+ } else {
+if (configFormat != _format && configFormat != FileFormat.AVRO) {
+ LOGGER.warn("Find format conflict in command line and config file,
use config in command line: {}", _format);
+}
}
}
@@ -291,7 +293,10 @@ public class CreateSegmentCommand extends
AbstractBaseAdminCommand implements Co
File[] files = dir.listFiles(new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
-return name.toLowerCase().endsWith(_format.toString().toLowerCase());
+if (_format != null) {
+ return name.toLowerCase().endsWith(_format.toString().toLowerCase());
+}
+return true;
}
});
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch createSegments created (now bdb2648)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch createSegments in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at bdb2648 Add support for custom record-readers in the create-segment tool This branch includes the following new commits: new bdb2648 Add support for custom record-readers in the create-segment tool The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Emit freshnessLag metric from broker for queries hitting consuming segments (#4229)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 06baad6 Emit freshnessLag metric from broker for queries hitting
consuming segments (#4229)
06baad6 is described below
commit 06baad647bfb04ac7c8c1796a7318edea47815d2
Author: Sunitha Beeram
AuthorDate: Tue May 21 22:21:10 2019 -0700
Emit freshnessLag metric from broker for queries hitting consuming segments
(#4229)
* Emit freshnessLag metric from broker for queries hitting consuming
segments
* Address review comments
* Minor formatting fixes
---
.../broker/requesthandler/BaseBrokerRequestHandler.java | 6 --
.../org/apache/pinot/common/metrics/BrokerTimer.java| 4 +++-
.../apache/pinot/common/response/BrokerResponse.java| 10 ++
.../common/response/broker/BrokerResponseNative.java| 6 +-
.../core/query/executor/ServerQueryExecutorV1Impl.java | 2 +-
.../pinot/core/query/reduce/BrokerReduceService.java| 16 ++--
.../pinot/core/query/scheduler/QueryScheduler.java | 17 -
7 files changed, 45 insertions(+), 16 deletions(-)
diff --git
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
index 7bc689f..89d3470 100644
---
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
+++
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
@@ -308,12 +308,14 @@ public abstract class BaseBrokerRequestHandler implements
BrokerRequestHandler {
if (_queryLogRateLimiter.tryAcquire() || forceLog(brokerResponse,
totalTimeMs)) {
// Table name might have been changed (with suffix _OFFLINE/_REALTIME
appended)
LOGGER.info(
- "RequestId:{}, table:{}, timeMs:{}, docs:{}/{}, entries:{}/{},
segments(queried/processed/matched):{}/{}/{} "
- + "servers:{}/{}, groupLimitReached:{}, exceptions:{},
serverStats:{}, query:{}", requestId,
+ "RequestId:{}, table:{}, timeMs:{}, docs:{}/{}, entries:{}/{},"
+ + " segments(queried/processed/matched/consuming):{}/{}/{}/{},
consumingFreshnessTimeMs:{},"
+ + " servers:{}/{}, groupLimitReached:{}, exceptions:{},
serverStats:{}, query:{}", requestId,
brokerRequest.getQuerySource().getTableName(), totalTimeMs,
brokerResponse.getNumDocsScanned(),
brokerResponse.getTotalDocs(),
brokerResponse.getNumEntriesScannedInFilter(),
brokerResponse.getNumEntriesScannedPostFilter(),
brokerResponse.getNumSegmentsQueried(),
brokerResponse.getNumSegmentsProcessed(),
brokerResponse.getNumSegmentsMatched(),
+ brokerResponse.getNumConsumingSegmentsQueried(),
brokerResponse.getMinConsumingFreshnessTimeMs(),
brokerResponse.getNumServersResponded(),
brokerResponse.getNumServersQueried(),
brokerResponse.isNumGroupsLimitReached(),
brokerResponse.getExceptionsSize(), serverStats.getServerStats(),
StringUtils.substring(query, 0, _queryLogLength));
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerTimer.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerTimer.java
index 694db60..6c98dab 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerTimer.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerTimer.java
@@ -27,7 +27,9 @@ import org.apache.pinot.common.Utils;
*/
public enum BrokerTimer implements AbstractMetrics.Timer {
ROUTING_TABLE_UPDATE_TIME(true),
- CLUSTER_CHANGE_QUEUE_TIME(true);
+ CLUSTER_CHANGE_QUEUE_TIME(true),
+ // metric tracking the freshness lag for consuming segments
+ FRESHNESS_LAG_MS(false);
private final String timerName;
private final boolean global;
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
b/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
index 72b95e2..ec2960e 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
@@ -98,6 +98,16 @@ public interface BrokerResponse {
long getNumSegmentsMatched();
/**
+ * Get number of consuming segments that were queried.
+ */
+ long getNumConsumingSegmentsQueried();
+
+ /**
+ * Get the minimum freshness timestamp across consuming segments that were
queried
+ */
+ long getMinConsumingFreshnessTimeMs();
+
+ /**
* Get total number of documents within the table hit.
*/
long getTotalDocs();
diff --
[incubator-pinot] branch consumingStats deleted (was 704b346)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch consumingStats in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 704b346 Minor formatting fixes The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch consumingStats updated (e6b9b59 -> 704b346)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch consumingStats in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from e6b9b59 Address review comments add 704b346 Minor formatting fixes No new revisions were added by this update. Summary of changes: .../pinot/common/response/broker/BrokerResponseNative.java| 3 +++ .../org/apache/pinot/core/query/scheduler/QueryScheduler.java | 11 ++- 2 files changed, 9 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch consumingStats updated (788e7cc -> e6b9b59)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch consumingStats in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 788e7cc Emit freshnessLag metric from broker for queries hitting consuming segments add e6b9b59 Address review comments No new revisions were added by this update. Summary of changes: .../requesthandler/BaseBrokerRequestHandler.java | 4 ++-- .../org/apache/pinot/common/metrics/BrokerMeter.java | 3 --- .../org/apache/pinot/common/metrics/BrokerTimer.java | 4 +++- .../common/response/broker/BrokerResponseNative.java | 2 ++ .../core/query/executor/ServerQueryExecutorV1Impl.java | 4 +--- .../pinot/core/query/reduce/BrokerReduceService.java | 18 +- 6 files changed, 13 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch consumingStats updated (a5066fb -> 788e7cc)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch consumingStats in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard a5066fb Emit freshnessLag metric from broker for queries hitting consuming segments new 788e7cc Emit freshnessLag metric from broker for queries hitting consuming segments This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (a5066fb) \ N -- N -- N refs/heads/consumingStats (788e7cc) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../main/java/org/apache/pinot/common/response/BrokerResponse.java | 6 ++ 1 file changed, 6 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Emit freshnessLag metric from broker for queries hitting consuming segments
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch consumingStats
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 788e7cc151e407d5e1c13f1da748b270e96d6aac
Author: Sunitha Beeram
AuthorDate: Tue May 21 10:55:51 2019 -0700
Emit freshnessLag metric from broker for queries hitting consuming segments
---
.../requesthandler/BaseBrokerRequestHandler.java | 6 --
.../org/apache/pinot/common/metrics/BrokerMeter.java | 3 +++
.../apache/pinot/common/response/BrokerResponse.java | 10 ++
.../common/response/broker/BrokerResponseNative.java | 1 -
.../query/executor/ServerQueryExecutorV1Impl.java| 4 +++-
.../pinot/core/query/reduce/BrokerReduceService.java | 20
.../pinot/core/query/scheduler/QueryScheduler.java | 12 +---
7 files changed, 45 insertions(+), 11 deletions(-)
diff --git
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
index 7bc689f..5f503dd 100644
---
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
+++
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
@@ -308,12 +308,14 @@ public abstract class BaseBrokerRequestHandler implements
BrokerRequestHandler {
if (_queryLogRateLimiter.tryAcquire() || forceLog(brokerResponse,
totalTimeMs)) {
// Table name might have been changed (with suffix _OFFLINE/_REALTIME
appended)
LOGGER.info(
- "RequestId:{}, table:{}, timeMs:{}, docs:{}/{}, entries:{}/{},
segments(queried/processed/matched):{}/{}/{} "
- + "servers:{}/{}, groupLimitReached:{}, exceptions:{},
serverStats:{}, query:{}", requestId,
+ "RequestId:{}, table:{}, timeMs:{}, docs:{}/{}, entries:{}/{},
segments(queried/processed/matched/consuming):{}/{}/{}/{},"
+ + " consumingFreshnessTimeMs:{},"
+ + " servers:{}/{}, groupLimitReached:{}, exceptions:{},
serverStats:{}, query:{}", requestId,
brokerRequest.getQuerySource().getTableName(), totalTimeMs,
brokerResponse.getNumDocsScanned(),
brokerResponse.getTotalDocs(),
brokerResponse.getNumEntriesScannedInFilter(),
brokerResponse.getNumEntriesScannedPostFilter(),
brokerResponse.getNumSegmentsQueried(),
brokerResponse.getNumSegmentsProcessed(),
brokerResponse.getNumSegmentsMatched(),
+ brokerResponse.getNumConsumingSegmentsQueried(),
brokerResponse.getMinConsumingFreshnessTimeMs(),
brokerResponse.getNumServersResponded(),
brokerResponse.getNumServersQueried(),
brokerResponse.isNumGroupsLimitReached(),
brokerResponse.getExceptionsSize(), serverStats.getServerStats(),
StringUtils.substring(query, 0, _queryLogLength));
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java
index 170e745..facd15f 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java
@@ -63,6 +63,9 @@ public enum BrokerMeter implements AbstractMetrics.Meter {
ENTRIES_SCANNED_IN_FILTER("documents", false),
ENTRIES_SCANNED_POST_FILTER("documents", false),
+ // metric tracking the freshness lag for consuming segments
+ FRESHNESS_LAG_MS("latency", false),
+
REQUEST_CONNECTION_TIMEOUTS("timeouts", false),
HELIX_ZOOKEEPER_RECONNECTS("reconnects", true),
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
b/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
index 72b95e2..ec2960e 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/response/BrokerResponse.java
@@ -98,6 +98,16 @@ public interface BrokerResponse {
long getNumSegmentsMatched();
/**
+ * Get number of consuming segments that were queried.
+ */
+ long getNumConsumingSegmentsQueried();
+
+ /**
+ * Get the minimum freshness timestamp across consuming segments that were
queried
+ */
+ long getMinConsumingFreshnessTimeMs();
+
+ /**
* Get total number of documents within the table hit.
*/
long getTotalDocs();
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
b/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
index 214d942..1ccedd6 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
+++
b/pi
[incubator-pinot] branch consumingStats created (now a5066fb)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch consumingStats in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at a5066fb Emit freshnessLag metric from broker for queries hitting consuming segments This branch includes the following new commits: new a5066fb Emit freshnessLag metric from broker for queries hitting consuming segments The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Track "freshness" timestamp across consuming segments (#3979)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 18d628b Track "freshness" timestamp across consuming segments (#3979)
18d628b is described below
commit 18d628b8fda8cf27c58854b64b8d4ee177c5334b
Author: Sunitha Beeram
AuthorDate: Wed May 8 21:47:58 2019 -0700
Track "freshness" timestamp across consuming segments (#3979)
* Track Indexed timestamp across consuming segments
* Fix when indexing time is updated
* Update based on PR#4025
* Fix tests
* Simplify interfaces
* Update interface annotation
* Address review comments
* Fix names in BrokerResponseNative; Rely on instanceof
* Address review comments
* Fix name
* Fix test
---
.../apache/pinot/common/metadata/RowMetadata.java | 30 ++
.../response/broker/BrokerResponseNative.java | 27 +++-
.../org/apache/pinot/common/utils/DataTable.java | 2 ++
.../realtime/HLRealtimeSegmentDataManager.java | 7 ++--
.../realtime/LLRealtimeSegmentDataManager.java | 7 ++--
.../core/indexsegment/mutable/MutableSegment.java | 7 ++--
.../indexsegment/mutable/MutableSegmentImpl.java | 23 +++---
.../query/executor/ServerQueryExecutorV1Impl.java | 37 +-
.../core/query/reduce/BrokerReduceService.java | 19 +++
.../impl/kafka/KafkaStreamLevelConsumer.java | 7 ++--
.../pinot/core/realtime/stream/MessageBatch.java | 7 ++--
.../core/realtime/stream/StreamLevelConsumer.java | 11 +--
.../core/realtime/stream/StreamMessageDecoder.java | 36 -
.../realtime/stream/StreamMessageMetadata.java | 25 +--
.../MutableSegmentImplAggregateMetricsTest.java| 2 +-
.../mutable/MutableSegmentImplTest.java| 15 -
.../tests/LLCRealtimeClusterIntegrationTest.java | 4 +++
.../realtime/provisioning/MemoryEstimator.java | 5 +--
18 files changed, 148 insertions(+), 123 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamMessageMetadata.java
b/pinot-common/src/main/java/org/apache/pinot/common/metadata/RowMetadata.java
similarity index 64%
copy from
pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamMessageMetadata.java
copy to
pinot-common/src/main/java/org/apache/pinot/common/metadata/RowMetadata.java
index 30c0859..4e67a05 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/realtime/stream/StreamMessageMetadata.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metadata/RowMetadata.java
@@ -16,35 +16,23 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.realtime.stream;
+package org.apache.pinot.common.metadata;
import org.apache.pinot.annotations.InterfaceAudience;
import org.apache.pinot.annotations.InterfaceStability;
/**
- * A class that provides metadata associated with the message of a stream, for
e.g.,
- * ingestion-timestamp of the message.
+ * A class that provides relevant row-level metadata for rows ingested into a
segment.
+ *
+ * Currently this is relevant for rows ingested into a mutable segment.
*/
@InterfaceAudience.Public
@InterfaceStability.Evolving
-public class StreamMessageMetadata {
-
- private long _ingestionTimestamp = Long.MIN_VALUE;
-
- public StreamMessageMetadata() {
-
- }
-
- public long getIngestionTimestamp() {
-return _ingestionTimestamp;
- }
-
- public void setIngestionTimestamp(long timestamp) {
-_ingestionTimestamp = timestamp;
- }
+public interface RowMetadata {
- public void reset() {
-_ingestionTimestamp = Long.MIN_VALUE;
- }
+ /**
+ * Return the ingestion timestamp of the row.
+ */
+ long getIngestionTimeMs();
}
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
b/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
index 94c6d46..214d942 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
@@ -39,7 +39,7 @@ import org.apache.pinot.common.utils.JsonUtils;
*
* Supports serialization via JSON.
*/
-@JsonPropertyOrder({"selectionResults", "aggregationResults", "exceptions",
"numServersQueried", "numServersResponded", "numSegmentsQueried",
"numSegmentsProcessed", "numSegmentsMatched", "numDocsScanned",
"numEntriesScannedInFilter", "numEntriesScannedPostFilter",
"numGroupsLim
[incubator-pinot] branch master updated: Randomly choose segments that need to be moved (#4191)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 29b918f Randomly choose segments that need to be moved (#4191)
29b918f is described below
commit 29b918f998c24ad8de4e2bc7ec63a7180494824e
Author: Sunitha Beeram
AuthorDate: Wed May 8 13:07:41 2019 -0700
Randomly choose segments that need to be moved (#4191)
---
.../helix/core/rebalance/ReplicaGroupRebalanceSegmentStrategy.java | 5 +
1 file changed, 5 insertions(+)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ReplicaGroupRebalanceSegmentStrategy.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ReplicaGroupRebalanceSegmentStrategy.java
index a75f334..9b7d8e3 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ReplicaGroupRebalanceSegmentStrategy.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ReplicaGroupRebalanceSegmentStrategy.java
@@ -20,6 +20,7 @@ package org.apache.pinot.controller.helix.core.rebalance;
import com.google.common.base.Function;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
@@ -454,6 +455,8 @@ public class ReplicaGroupRebalanceSegmentStrategy
implements RebalanceSegmentStr
// Remove segments from servers that has more segments
for (String server : serversInReplicaGroup) {
LinkedList segmentsInServer = serverToSegments.get(server);
+ // randomize the segments to be moved
+ Collections.shuffle(segmentsInServer);
int segmentToMove = numSegmentsPerServer - segmentsInServer.size();
if (segmentToMove < 0) {
// Server has more segments than needed, remove segments from this
server
@@ -463,6 +466,8 @@ public class ReplicaGroupRebalanceSegmentStrategy
implements RebalanceSegmentStr
}
}
+// randomize the segments to be added
+Collections.shuffle(segmentsToAdd);
// Add segments to servers that has less segments than numSegmentsPerServer
for (String server : serversInReplicaGroup) {
LinkedList segmentsInServer = serverToSegments.get(server);
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Update Audience and Stability annotations for basic classes (#4169)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 71a63c8 Update Audience and Stability annotations for basic classes
(#4169)
71a63c8 is described below
commit 71a63c8fcf927fcb335f3ebc1988434eda0b8a38
Author: Sunitha Beeram
AuthorDate: Fri Apr 26 16:04:19 2019 -0700
Update Audience and Stability annotations for basic classes (#4169)
* Update annotations
* Remove annotations for FSFactory class
* Remove annotation for BrokerRequestHandler
* Reformat code
---
.../src/main/java/org/apache/pinot/broker/api/AccessControl.java| 4
pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFS.java | 4
.../java/org/apache/pinot/controller/api/access/AccessControl.java | 4
.../apache/pinot/controller/api/access/AccessControlFactory.java| 6 ++
4 files changed, 18 insertions(+)
diff --git
a/pinot-broker/src/main/java/org/apache/pinot/broker/api/AccessControl.java
b/pinot-broker/src/main/java/org/apache/pinot/broker/api/AccessControl.java
index 70d6d24..86b6551 100644
--- a/pinot-broker/src/main/java/org/apache/pinot/broker/api/AccessControl.java
+++ b/pinot-broker/src/main/java/org/apache/pinot/broker/api/AccessControl.java
@@ -18,9 +18,13 @@
*/
package org.apache.pinot.broker.api;
+import org.apache.pinot.annotations.InterfaceAudience;
+import org.apache.pinot.annotations.InterfaceStability;
import org.apache.pinot.common.request.BrokerRequest;
+@InterfaceAudience.Public
+@InterfaceStability.Stable
public interface AccessControl {
boolean hasAccess(RequesterIdentity requesterIdentity, BrokerRequest
brokerRequest);
}
diff --git
a/pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFS.java
b/pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFS.java
index 8032014..4014441 100644
--- a/pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFS.java
+++ b/pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFS.java
@@ -24,6 +24,8 @@ import java.io.IOException;
import java.net.URI;
import java.nio.file.Paths;
import org.apache.commons.configuration.Configuration;
+import org.apache.pinot.annotations.InterfaceAudience;
+import org.apache.pinot.annotations.InterfaceStability;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -38,6 +40,8 @@ import org.slf4j.LoggerFactory;
*The contract that developers have to adhere to will be simpler.
* Please read the method level docs carefully to note the exceptions while
using the APIs.
*/
+@InterfaceAudience.Public
+@InterfaceStability.Stable
public abstract class PinotFS implements Closeable {
private static final Logger LOGGER = LoggerFactory.getLogger(PinotFS.class);
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControl.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControl.java
index 7e8b161..37b3f3b 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControl.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControl.java
@@ -19,8 +19,12 @@
package org.apache.pinot.controller.api.access;
import javax.ws.rs.core.HttpHeaders;
+import org.apache.pinot.annotations.InterfaceAudience;
+import org.apache.pinot.annotations.InterfaceStability;
+@InterfaceAudience.Public
+@InterfaceStability.Stable
public interface AccessControl {
/**
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControlFactory.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControlFactory.java
index 12020f1..8463666 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControlFactory.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControlFactory.java
@@ -18,6 +18,12 @@
*/
package org.apache.pinot.controller.api.access;
+import org.apache.pinot.annotations.InterfaceAudience;
+import org.apache.pinot.annotations.InterfaceStability;
+
+
+@InterfaceAudience.Public
+@InterfaceStability.Stable
public interface AccessControlFactory {
AccessControl create();
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch Helix084-ReleaseApr25 created (now 0e612e0)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch Helix084-ReleaseApr25 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 0e612e0 Prepare for internal release This branch includes the following new commits: new 0e612e0 Prepare for internal release The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Prepare for internal release
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch Helix084-ReleaseApr25 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 0e612e05d25ae05722a70b1f04c916f496d6ce3a Author: Sunitha Beeram AuthorDate: Thu Apr 25 11:47:43 2019 -0700 Prepare for internal release --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 9dc5612..58acfd0 100644 --- a/pom.xml +++ b/pom.xml @@ -117,7 +117,7 @@ 1.7.6 1.8.0 -0.8.2 +0.8.4 0.9.0.1 0.7 - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Update documentation to reflect the new dependency structure (#4163)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new e8a620f Update documentation to reflect the new dependency structure (#4163) e8a620f is described below commit e8a620f2f7824594987d365d74145098cc1f502e Author: Sunitha Beeram AuthorDate: Wed Apr 24 20:27:32 2019 -0700 Update documentation to reflect the new dependency structure (#4163) --- docs/code_modules.rst| 9 +++-- docs/img/PinotController.png | Bin 47991 -> 61419 bytes docs/img/PinotFoundation.png | Bin 44122 -> 29973 bytes docs/img/PinotServer.png | Bin 55062 -> 73944 bytes 4 files changed, 3 insertions(+), 6 deletions(-) diff --git a/docs/code_modules.rst b/docs/code_modules.rst index cf439cb..0ab26b9 100644 --- a/docs/code_modules.rst +++ b/docs/code_modules.rst @@ -72,12 +72,6 @@ The following figure provides a high-level overview of the foundational Pinot mo .. figure:: img/PinotFoundation.png :scale: 50 % -pinot-filesystem - -``pinot-filesystem`` provides abstractions for working with ``segments`` on local or remote filesystems. This module -allows for users to plugin filesystems specific to their usecase. Extensions to the base ``PinotFS`` should ideally be -housed in their specific modules so as not pull in unnecessary dependencies for all users. - pinot-common ``pinot-common`` provides classes common to Pinot components. Some key classes you will find here are: @@ -90,6 +84,9 @@ pinot-common * ``pql.parsers``: Code to compile PQL strings into corresponding AbstractSyntaxTrees (AST). * ``request``: Autogenerated thrift classes representing various parts of PQL requests. * ``response``: Definitions of response format returned by the Broker. +* ``filesystem`` provides abstractions for working with ``segments`` on local or remote filesystems. This module +allows for users to plugin filesystems specific to their usecase. Extensions to the base ``PinotFS`` should ideally be +housed in their specific modules so as not pull in unnecessary dependencies for all users. pinot-transport ^^^ diff --git a/docs/img/PinotController.png b/docs/img/PinotController.png index 63a658b..f62d5ed 100644 Binary files a/docs/img/PinotController.png and b/docs/img/PinotController.png differ diff --git a/docs/img/PinotFoundation.png b/docs/img/PinotFoundation.png index 91a85a4..c2547d4 100644 Binary files a/docs/img/PinotFoundation.png and b/docs/img/PinotFoundation.png differ diff --git a/docs/img/PinotServer.png b/docs/img/PinotServer.png index 38ac17e..9cbb2cb 100644 Binary files a/docs/img/PinotServer.png and b/docs/img/PinotServer.png differ - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Remove filesystem module from pinot-distribution (#4165)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 2401249 Remove filesystem module from pinot-distribution (#4165)
2401249 is described below
commit 2401249ff72cc6eef4f8ff3c68754ce242bbbfbd
Author: Sunitha Beeram
AuthorDate: Wed Apr 24 20:27:55 2019 -0700
Remove filesystem module from pinot-distribution (#4165)
---
pinot-controller/pom.xml | 4
pinot-distribution/pom.xml | 4
pom.xml| 5 -
3 files changed, 13 deletions(-)
diff --git a/pinot-controller/pom.xml b/pinot-controller/pom.xml
index 75ac65f..6876c22 100644
--- a/pinot-controller/pom.xml
+++ b/pinot-controller/pom.xml
@@ -220,10 +220,6 @@
-
- org.apache.pinot
- pinot-filesystem
-
diff --git a/pinot-distribution/pom.xml b/pinot-distribution/pom.xml
index c391cdc..321a131 100644
--- a/pinot-distribution/pom.xml
+++ b/pinot-distribution/pom.xml
@@ -66,10 +66,6 @@
org.apache.pinot
- pinot-filesystem
-
-
- org.apache.pinot
pinot-hadoop-filesystem
diff --git a/pom.xml b/pom.xml
index ec083f3..9dc5612 100644
--- a/pom.xml
+++ b/pom.xml
@@ -274,11 +274,6 @@
org.apache.pinot
-pinot-filesystem
-${project.version}
-
-
-org.apache.pinot
pinot-hadoop-filesystem
${project.version}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Move pinot-filesystem into pinot-common (#4155)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 32b505c Move pinot-filesystem into pinot-common (#4155)
32b505c is described below
commit 32b505ce9d8d348238cbeef4c43b102af65e442e
Author: Sunitha Beeram
AuthorDate: Wed Apr 24 15:43:09 2019 -0700
Move pinot-filesystem into pinot-common (#4155)
---
pinot-azure-filesystem/pom.xml | 6 --
pinot-common/pom.xml | 4 --
.../org/apache/pinot/filesystem/LocalPinotFS.java | 0
.../java/org/apache/pinot/filesystem/PinotFS.java | 0
.../apache/pinot/filesystem/PinotFSFactory.java| 0
.../apache/pinot/filesystem/LocalPinotFSTest.java | 0
.../pinot/filesystem/PinotFSFactoryTest.java | 0
pinot-filesystem/pom.xml | 69 --
pinot-hadoop-filesystem/pom.xml| 6 --
pom.xml| 1 -
10 files changed, 86 deletions(-)
diff --git a/pinot-azure-filesystem/pom.xml b/pinot-azure-filesystem/pom.xml
index d18a6fa..53acd8d 100644
--- a/pinot-azure-filesystem/pom.xml
+++ b/pinot-azure-filesystem/pom.xml
@@ -50,12 +50,6 @@
org.apache.pinot
- pinot-filesystem
- 0.2.0-SNAPSHOT
- compile
-
-
- org.apache.pinot
pinot-common
diff --git a/pinot-common/pom.xml b/pinot-common/pom.xml
index 55485a7..0b12021 100644
--- a/pinot-common/pom.xml
+++ b/pinot-common/pom.xml
@@ -301,10 +301,6 @@
zookeeper
- org.apache.pinot
- pinot-filesystem
-
-
javax.servlet
javax.servlet-api
compile
diff --git
a/pinot-filesystem/src/main/java/org/apache/pinot/filesystem/LocalPinotFS.java
b/pinot-common/src/main/java/org/apache/pinot/filesystem/LocalPinotFS.java
similarity index 100%
rename from
pinot-filesystem/src/main/java/org/apache/pinot/filesystem/LocalPinotFS.java
rename to
pinot-common/src/main/java/org/apache/pinot/filesystem/LocalPinotFS.java
diff --git
a/pinot-filesystem/src/main/java/org/apache/pinot/filesystem/PinotFS.java
b/pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFS.java
similarity index 100%
rename from
pinot-filesystem/src/main/java/org/apache/pinot/filesystem/PinotFS.java
rename to pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFS.java
diff --git
a/pinot-filesystem/src/main/java/org/apache/pinot/filesystem/PinotFSFactory.java
b/pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFSFactory.java
similarity index 100%
rename from
pinot-filesystem/src/main/java/org/apache/pinot/filesystem/PinotFSFactory.java
rename to
pinot-common/src/main/java/org/apache/pinot/filesystem/PinotFSFactory.java
diff --git
a/pinot-filesystem/src/test/java/org/apache/pinot/filesystem/LocalPinotFSTest.java
b/pinot-common/src/test/java/org/apache/pinot/filesystem/LocalPinotFSTest.java
similarity index 100%
rename from
pinot-filesystem/src/test/java/org/apache/pinot/filesystem/LocalPinotFSTest.java
rename to
pinot-common/src/test/java/org/apache/pinot/filesystem/LocalPinotFSTest.java
diff --git
a/pinot-filesystem/src/test/java/org/apache/pinot/filesystem/PinotFSFactoryTest.java
b/pinot-common/src/test/java/org/apache/pinot/filesystem/PinotFSFactoryTest.java
similarity index 100%
rename from
pinot-filesystem/src/test/java/org/apache/pinot/filesystem/PinotFSFactoryTest.java
rename to
pinot-common/src/test/java/org/apache/pinot/filesystem/PinotFSFactoryTest.java
diff --git a/pinot-filesystem/pom.xml b/pinot-filesystem/pom.xml
deleted file mode 100644
index 047755a..000
--- a/pinot-filesystem/pom.xml
+++ /dev/null
@@ -1,69 +0,0 @@
-
-
-http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
- 4.0.0
-
-pinot
-org.apache.pinot
-0.2.0-SNAPSHOT
-
- pinot-filesystem
- Pinot Filesystem
- https://pinot.apache.org/
-
-${basedir}/..
-
-
-
-
-org.apache.maven.plugins
-maven-enforcer-plugin
-
-
-
-
-
- org.testng
- testng
- test
-
-
- org.mockito
- mockito-core
- test
-
-
- commons-io
- commons-io
-
-
- commons-configuration
- commons-configuration
- 1.6
-
-
- org.slf4j
- slf4j-api
-
-
-
diff --git a/pinot-hadoop-filesystem/pom.xml b/pinot-hadoop-filesystem/pom.xml
index df4f7f5..28ce7dc 100644
--- a/pinot-hadoop-filesystem/pom.xml
+++ b/pinot-hadoop-filesystem/pom.xml
@@ -73,12 +73,6 @@
org.apache.pinot
- pinot-filesystem
- ${project.version}
- compile
-
-
- org.apache.pinot
[incubator-pinot] branch master updated: Extend stream interfaces to expose message metadata like ingestion timestamp (#4025)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 3a1e386 Extend stream interfaces to expose message metadata like
ingestion timestamp (#4025)
3a1e386 is described below
commit 3a1e386e4ea96d1b8629a89e065eaac274553b6e
Author: Sunitha Beeram
AuthorDate: Tue Apr 9 10:49:24 2019 -0700
Extend stream interfaces to expose message metadata like ingestion
timestamp (#4025)
* Extend stream interfaces to expose message metadata like ingestion
timestamp
* Update segment metadata interface
* Expose timestamps through segment-metadata instead of IndexSegment
* Fix tracking
* Update MessageBatch API to take stream-metadata object
* Address review comments
* Address review comments
* Minor fixes
* Update MutableSegment interface with just one index method
* Address review comments
---
.../pinot/common/segment/SegmentMetadata.java | 16 +++
.../realtime/HLRealtimeSegmentDataManager.java | 7 ++-
.../realtime/LLRealtimeSegmentDataManager.java | 11 -
.../core/indexsegment/mutable/MutableSegment.java | 6 ++-
.../indexsegment/mutable/MutableSegmentImpl.java | 33 --
.../impl/kafka/KafkaStreamLevelConsumer.java | 8 +++-
.../pinot/core/realtime/stream/MessageBatch.java | 10
.../core/realtime/stream/StreamLevelConsumer.java | 10
.../core/realtime/stream/StreamMessageDecoder.java | 36 +++
...velConsumer.java => StreamMessageMetadata.java} | 53 +-
.../core/segment/index/SegmentMetadataImpl.java| 14 ++
.../MutableSegmentImplAggregateMetricsTest.java| 4 +-
.../mutable/MutableSegmentImplTest.java| 4 +-
.../realtime/provisioning/MemoryEstimator.java | 6 ++-
14 files changed, 174 insertions(+), 44 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/segment/SegmentMetadata.java
b/pinot-common/src/main/java/org/apache/pinot/common/segment/SegmentMetadata.java
index 68d6ffb..9368ee2 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/segment/SegmentMetadata.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/segment/SegmentMetadata.java
@@ -73,6 +73,22 @@ public interface SegmentMetadata {
*/
long getRefreshTime();
+ /**
+ * Return the last time a record was indexed in this segment. Applicable for
MutableSegments.
+ *
+ * @return time when the last record was indexed
+ */
+ long getLastIndexedTimestamp();
+
+ /**
+ * Return the latest ingestion timestamp associated with the records indexed
in this segment.
+ * Applicable for MutableSegments.
+ *
+ * @return latest timestamp associated with indexed records
+ * Long.MIN_VALUE if the stream doesn't provide a
timestamp
+ */
+ long getLatestIngestionTimestamp();
+
boolean hasDictionary(String columnName);
boolean hasStarTree();
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
index 745115b..a8e0bdd 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
@@ -51,6 +51,7 @@ import org.apache.pinot.core.realtime.stream.StreamConfig;
import org.apache.pinot.core.realtime.stream.StreamConsumerFactory;
import org.apache.pinot.core.realtime.stream.StreamConsumerFactoryProvider;
import org.apache.pinot.core.realtime.stream.StreamLevelConsumer;
+import org.apache.pinot.core.realtime.stream.StreamMessageMetadata;
import org.apache.pinot.core.segment.index.loader.IndexLoadingConfig;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
@@ -211,15 +212,17 @@ public class HLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
int numRowsErrored = 0;
GenericRow consumedRow = null;
+StreamMessageMetadata messageMetadata = new StreamMessageMetadata();
do {
try {
consumedRow = GenericRow.createOrReuseRow(consumedRow);
-consumedRow = _streamLevelConsumer.next(consumedRow);
+messageMetadata.reset();
+consumedRow = _streamLevelConsumer.next(consumedRow,
messageMetadata);
if (consumedRow != null) {
GenericRow transformedRow =
_recordTransformer.transform(consumedRow);
if (transformedRow != null) {
-notFull = realtimeSegment.index(transformedRow);
+notFull = realtimeSegment.index(transformedRow,
messageMetad
[incubator-pinot] branch master updated: Introduce interface stability and audience annotations (#4063)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 3a9625b Introduce interface stability and audience annotations (#4063)
3a9625b is described below
commit 3a9625bec5c5e986004f61246a80725b0c9bf4a3
Author: Sunitha Beeram
AuthorDate: Thu Apr 4 12:29:42 2019 -0700
Introduce interface stability and audience annotations (#4063)
* Introduce interface stability and audience annotations
* Address review comments
* Address review comments
* Address review comments
---
.../pinot/annotations/InterfaceAudience.java | 73 ++
.../pinot/annotations/InterfaceStability.java | 70 +
.../pinot/core/indexsegment/IndexSegment.java | 3 +-
.../pinot/core/realtime/stream/MessageBatch.java | 6 ++
.../realtime/stream/PartitionLevelConsumer.java| 4 ++
.../core/realtime/stream/StreamLevelConsumer.java | 4 ++
.../core/realtime/stream/StreamMessageDecoder.java | 4 ++
.../realtime/stream/StreamMetadataProvider.java| 4 ++
8 files changed, 167 insertions(+), 1 deletion(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/annotations/InterfaceAudience.java
b/pinot-common/src/main/java/org/apache/pinot/annotations/InterfaceAudience.java
new file mode 100644
index 000..865bdf6
--- /dev/null
+++
b/pinot-common/src/main/java/org/apache/pinot/annotations/InterfaceAudience.java
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.annotations;
+
+import java.lang.annotation.Documented;
+
+
+/**
+ * Annotation to inform users of a package, class or method's intended
audience.
+ *
+ * Currently the audience can be {@link
org.apache.pinot.annotations.InterfaceAudience.Public},
+ * {@link org.apache.pinot.annotations.InterfaceAudience.LimitedPrivate} or
+ * {@link org.apache.pinot.annotations.InterfaceAudience.Private}.
+ *
+ *
+ * Public classes that are not marked with this annotation must be
+ * considered by default as {@link
org.apache.pinot.annotations.InterfaceAudience.Private}.
+ *
+ * External application developers depending on Pinot must only use
classes that are marked
+ * {@link org.apache.pinot.annotations.InterfaceAudience.Public}. Do not
depend on classes without an
+ * explicit InterfaceAudience.Public annotation as these classes
+ * could be removed or change in incompatible ways.
+ *
+ * Methods may have a different annotation that it is more restrictive
+ * compared to the audience classification of the class. Example: A class
+ * might be {@link org.apache.pinot.annotations.InterfaceAudience.Public},
+ * but a method may be {@link
org.apache.pinot.annotations.InterfaceAudience.LimitedPrivate}
+ *
+ *
+ * The annotation is borrowed from a similar Apache Hadoop annotation.
+ */
+@InterfaceAudience.Public
+@InterfaceStability.Evolving
+public class InterfaceAudience {
+ /**
+ * Intended for use by any project or application.
+ */
+ @Documented
+ public @interface Public {};
+
+ /**
+ * Intended only for the project(s) specified in the annotation.
+ * Reserved for future use.
+ */
+ @Documented
+ public @interface LimitedPrivate {
+String[] value();
+ };
+
+ /**
+ * Intended for use only within Pinot itself.
+ */
+ @Documented
+ public @interface Private {};
+
+ private InterfaceAudience() {} // Audience can't exist on its own
+}
\ No newline at end of file
diff --git
a/pinot-common/src/main/java/org/apache/pinot/annotations/InterfaceStability.java
b/pinot-common/src/main/java/org/apache/pinot/annotations/InterfaceStability.java
new file mode 100644
index 000..26eeda3
--- /dev/null
+++
b/pinot-common/src/main/java/org/apache/pinot/annotations/InterfaceStability.java
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The AS
[incubator-pinot] branch master updated: Handle exceptions during teardown (#4055)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new aaef725 Handle exceptions during teardown (#4055)
aaef725 is described below
commit aaef725898fa8898ba63b1a8544fc558ed1b2969
Author: Sunitha Beeram
AuthorDate: Tue Apr 2 10:36:20 2019 -0700
Handle exceptions during teardown (#4055)
*
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ClusterTest.java
* Handle exceptions during teardown
---
.../apache/pinot/integration/tests/ClusterTest.java | 19 ---
1 file changed, 16 insertions(+), 3 deletions(-)
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ClusterTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ClusterTest.java
index 9dd75c2..f677be3 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ClusterTest.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ClusterTest.java
@@ -81,6 +81,7 @@ import org.testng.Assert;
* Base class for integration tests that involve a complete Pinot cluster.
*/
public abstract class ClusterTest extends ControllerTest {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(ClusterTest.class);
private static final Random RANDOM = new Random();
private static final int DEFAULT_BROKER_PORT = 18099;
@@ -218,20 +219,32 @@ public abstract class ClusterTest extends ControllerTest {
protected void stopBroker() {
for (HelixBrokerStarter brokerStarter : _brokerStarters) {
- BrokerTestUtils.stopBroker(brokerStarter);
+ try {
+BrokerTestUtils.stopBroker(brokerStarter);
+ } catch (Exception e) {
+LOGGER.error("Encountered exception while stopping broker {}",
e.getMessage());
+ }
}
}
protected void stopServer() {
for (HelixServerStarter helixServerStarter : _serverStarters) {
- helixServerStarter.stop();
+ try {
+helixServerStarter.stop();
+ } catch (Exception e) {
+LOGGER.error("Encountered exception while stopping server {}",
e.getMessage());
+ }
}
FileUtils.deleteQuietly(new File(Server.DEFAULT_INSTANCE_BASE_DIR));
}
protected void stopMinion() {
for (MinionStarter minionStarter : _minionStarters) {
- minionStarter.stop();
+ try {
+minionStarter.stop();
+ } catch (Exception e) {
+LOGGER.error("Encountered exception while stopping minion {}",
e.getMessage());
+ }
}
FileUtils.deleteQuietly(new File(Minion.DEFAULT_INSTANCE_BASE_DIR));
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Set processingException when all queried segments cannot be acquired (#3942)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new d4f2ece Set processingException when all queried segments cannot be
acquired (#3942)
d4f2ece is described below
commit d4f2ecef660ab1d4efa9696a53b0623aac867c3f
Author: Sunitha Beeram
AuthorDate: Fri Mar 22 09:33:09 2019 -0700
Set processingException when all queried segments cannot be acquired (#3942)
* Set processingException when all queried segments cannot be acquired
* Minor fixes
* Address review comments
* Disable setting processing exception until metrics are validated
* Address review comments
* Address review comments
* Address review comments
* Address review comments
* Address review comments: Make InstanceDataManager symmetric with respect
to tracking segments added and deleted
---
.../pinot/common/exception/QueryException.java | 2 +
.../core/data/manager/BaseTableDataManager.java| 48 +
.../core/data/manager/InstanceDataManager.java | 14 ++
.../pinot/core/data/manager/TableDataManager.java | 15 +++
.../query/executor/ServerQueryExecutorV1Impl.java | 33 +-
.../data/manager/BaseTableDataManagerTest.java | 5 +++
.../pinot/query/executor/QueryExecutorTest.java| 50 --
.../starter/helix/HelixInstanceDataManager.java| 16 +++
.../SegmentOnlineOfflineStateModelFactory.java | 5 +++
9 files changed, 175 insertions(+), 13 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
b/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
index e0ed903..6c45897 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
@@ -41,6 +41,7 @@ public class QueryException {
public static final int SEGMENT_PLAN_EXECUTION_ERROR_CODE = 160;
public static final int COMBINE_SEGMENT_PLAN_TIMEOUT_ERROR_CODE = 170;
public static final int ACCESS_DENIED_ERROR_CODE = 180;
+ public static final int SEGMENTS_MISSING_ERROR_CODE = 190;
public static final int QUERY_EXECUTION_ERROR_CODE = 200;
// TODO: Handle these errors in broker
public static final int SERVER_SHUTTING_DOWN_ERROR_CODE = 210;
@@ -96,6 +97,7 @@ public class QueryException {
public static final ProcessingException QUERY_VALIDATION_ERROR = new
ProcessingException(QUERY_VALIDATION_ERROR_CODE);
public static final ProcessingException UNKNOWN_ERROR = new
ProcessingException(UNKNOWN_ERROR_CODE);
public static final ProcessingException QUOTA_EXCEEDED_ERROR = new
ProcessingException(TOO_MANY_REQUESTS_ERROR_CODE);
+ public static final ProcessingException SEGMENTS_MISSING_ERROR = new
ProcessingException(SEGMENTS_MISSING_ERROR_CODE);
static {
JSON_PARSING_ERROR.setMessage("JsonParsingError");
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
index a456691..3f074a1 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
@@ -19,10 +19,13 @@
package org.apache.pinot.core.data.manager;
import com.google.common.base.Preconditions;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
import javax.annotation.Nonnull;
import javax.annotation.concurrent.ThreadSafe;
import org.apache.helix.ZNRecord;
@@ -42,9 +45,13 @@ import org.slf4j.LoggerFactory;
@ThreadSafe
public abstract class BaseTableDataManager implements TableDataManager {
private static final Logger LOGGER =
LoggerFactory.getLogger(BaseTableDataManager.class);
+ // cache deleted segment names for utmost this duration
+ private static final int MAX_CACHE_DURATION_SEC = 6 * 3600; // 6 hours
protected final ConcurrentHashMap
_segmentDataManagerMap = new ConcurrentHashMap<>();
+ protected Cache _deletedSegmentsCache;
+
protected TableDataManagerConfig _tableDataManagerConfig;
protected String _instanceId;
protected ZkHelixPropertyStore _propertyStore;
@@ -59,6 +66,7 @@ public abstract class BaseTableDataManager implements
TableDataManager {
@Nonnull ZkHelixPropertyStore propertyStore, @Nonnull
ServerMetrics serverMetrics) {
LOGGER.info("Initializing table data manager for table: {}",
tableData
[incubator-pinot] branch master updated: Update managing pinot doc (#3991)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 205ec50 Update managing pinot doc (#3991) 205ec50 is described below commit 205ec5059cdf07dffc44355660642412bdbf3db5 Author: Jialiang Li AuthorDate: Wed Mar 20 11:09:24 2019 -0700 Update managing pinot doc (#3991) --- docs/management_api.rst | 21 ++--- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/management_api.rst b/docs/management_api.rst index e3f3c9d..9468bd1 100644 --- a/docs/management_api.rst +++ b/docs/management_api.rst @@ -26,7 +26,7 @@ Pinot Management Console There is a REST API which allows management of tables, tenants, segments and schemas. It can be accessed by going to -``http://[controller host]/help`` which offers a web UI to do these tasks, as well as document the REST API. The below +``http://[controller_host]/help`` which offers a web UI to do these tasks, as well as document the REST API. The below is the screenshot of the console. .. figure:: img/pinot-console.png @@ -43,17 +43,16 @@ To rebalance segments of a table across servers: pinot-admin.sh -- -It can be used instead of the ``pinot-admin.sh`` commands to automate the creation of tables and tenants. The script -can be generated by running ``mvn install package -DskipTests -Pbin-dist`` in the directory -in which you checked out Pinot. +``pinot-admin.sh`` is another way of managing Pinot cluster. This script can be generated by running +``mvn install package -DskipTests -Pbin-dist`` in the directory in which you checked out Pinot. For example, to create a pinot segment: .. code-block:: none - $ ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh CreateSegment -dataDir /Users/jlli/Desktop/test/ -format CSV -outDir /Users/jlli/Desktop/test2/ -tableName baseballStats -segmentName baseballStats_data -overwrite -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json - Executing command: CreateSegment -generatorConfigFile null -dataDir /Users/jlli/Desktop/test/ -format CSV -outDir /Users/jlli/Desktop/test2/ -overwrite true -tableName baseballStats -segmentName baseballStats_data -timeColumnName null -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json -readerConfigFile null -enableStarTreeIndex false -starTreeIndexSpecFile null -hllSize 9 - [...] - Accepted files: [/Users/jlli/Desktop/test/baseballStats_data.csv] + $ ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh CreateSegment -dataDir /Users/host1/Desktop/test/ -format CSV -outDir /Users/host1/Desktop/test2/ -tableName baseballStats -segmentName baseballStats_data -overwrite -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json + Executing command: CreateSegment -generatorConfigFile null -dataDir /Users/host1/Desktop/test/ -format CSV -outDir /Users/host1/Desktop/test2/ -overwrite true -tableName baseballStats -segmentName baseballStats_data -timeColumnName null -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json -readerConfigFile null -enableStarTreeIndex false -starTreeIndexSpecFile null -hllSize 9 [...] + Accepted files: [/Users/host1/Desktop/test/baseballStats_data.csv] Finished building StatsCollector! Collected stats for 97889 documents Created dictionary for INT column: homeRuns with cardinality: 67, range: 0 to 73 @@ -84,9 +83,9 @@ For example, to create a pinot segment: Start building IndexCreator! Finished records indexing in IndexCreator! Finished segment seal! - Converting segment: /Users/jlli/Desktop/test2/baseballStats_data_0 to v3 format - v3 segment location for segment: baseballStats_data_0 is /Users/jlli/Desktop/test2/baseballStats_data_0/v3 - Deleting files in v1 segment directory: /Users/jlli/Desktop/test2/baseballStats_data_0 + Converting segment: /Users/host1/Desktop/test2/baseballStats_data_0 to v3 format + v3 segment location for segment: baseballStats_data_0 is /Users/host1/Desktop/test2/baseballStats_data_0/v3 + Deleting files in v1 segment directory: /Users/host1/Desktop/test2/baseballStats_data_0 Driver, record read time : 369 Driver, stats collector time : 0 Driver, indexing time : 373 @@ -95,5 +94,5 @@ To query
[incubator-pinot] branch management-api-doc deleted (was a340bdf)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch management-api-doc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was a340bdf Update managing pinot doc The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Add documentation (#3986)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 59fd4aa Add documentation (#3986) 59fd4aa is described below commit 59fd4aab4480b6c62ba0bfe61f1a976d0ca31221 Author: Jialiang Li AuthorDate: Wed Mar 20 06:47:45 2019 -0700 Add documentation (#3986) --- docs/img/generate-segment.png | Bin 218597 -> 0 bytes docs/img/list-schemas.png | Bin 8952 -> 247946 bytes docs/img/pinot-console.png| Bin 0 -> 157310 bytes docs/img/query-table.png | Bin 35914 -> 0 bytes docs/img/rebalance-table.png | Bin 0 -> 164989 bytes docs/img/upload-segment.png | Bin 13944 -> 0 bytes docs/management_api.rst | 68 +++--- 7 files changed, 58 insertions(+), 10 deletions(-) diff --git a/docs/img/generate-segment.png b/docs/img/generate-segment.png deleted file mode 100644 index 5848781..000 Binary files a/docs/img/generate-segment.png and /dev/null differ diff --git a/docs/img/list-schemas.png b/docs/img/list-schemas.png index 9b00855..3bd0685 100644 Binary files a/docs/img/list-schemas.png and b/docs/img/list-schemas.png differ diff --git a/docs/img/pinot-console.png b/docs/img/pinot-console.png new file mode 100644 index 000..f73405b Binary files /dev/null and b/docs/img/pinot-console.png differ diff --git a/docs/img/query-table.png b/docs/img/query-table.png deleted file mode 100644 index 5859f2b..000 Binary files a/docs/img/query-table.png and /dev/null differ diff --git a/docs/img/rebalance-table.png b/docs/img/rebalance-table.png new file mode 100644 index 000..65de953 Binary files /dev/null and b/docs/img/rebalance-table.png differ diff --git a/docs/img/upload-segment.png b/docs/img/upload-segment.png deleted file mode 100644 index edd348d..000 Binary files a/docs/img/upload-segment.png and /dev/null differ diff --git a/docs/management_api.rst b/docs/management_api.rst index 5a7a0e4..e3f3c9d 100644 --- a/docs/management_api.rst +++ b/docs/management_api.rst @@ -26,26 +26,74 @@ Pinot Management Console There is a REST API which allows management of tables, tenants, segments and schemas. It can be accessed by going to -``http://[controller host]/help`` which offers a web UI to do these tasks, as well as document the REST API. +``http://[controller host]/help`` which offers a web UI to do these tasks, as well as document the REST API. The below +is the screenshot of the console. -For example, list all the schema within Pinot cluster: + .. figure:: img/pinot-console.png + +For example, to list all the schemas within Pinot cluster: .. figure:: img/list-schemas.png -Upload a pinot segment: +To rebalance segments of a table across servers: - .. figure:: img/upload-segment.png + .. figure:: img/rebalance-table.png -Pinot-admin.sh +pinot-admin.sh -- -It can be used instead of the ``pinot-admin.sh`` commands to automate the creation of tables and tenants. +It can be used instead of the ``pinot-admin.sh`` commands to automate the creation of tables and tenants. The script +can be generated by running ``mvn install package -DskipTests -Pbin-dist`` in the directory +in which you checked out Pinot. + +For example, to create a pinot segment: + +.. code-block:: none -For example, create a pinot segment: + $ ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/bin/pinot-admin.sh CreateSegment -dataDir /Users/jlli/Desktop/test/ -format CSV -outDir /Users/jlli/Desktop/test2/ -tableName baseballStats -segmentName baseballStats_data -overwrite -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json + Executing command: CreateSegment -generatorConfigFile null -dataDir /Users/jlli/Desktop/test/ -format CSV -outDir /Users/jlli/Desktop/test2/ -overwrite true -tableName baseballStats -segmentName baseballStats_data -timeColumnName null -schemaFile ./pinot-distribution/target/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/apache-pinot-incubating-0.1.0-SNAPSHOT-bin/sample_data/baseballStats_schema.json -readerConfigFile null -enableStarTreeIndex false -starTreeIndexSpecFile null -hllSize 9 - [...] + Accepted files: [/Users/jlli/Desktop/test/baseballStats_data.csv] + Finished building StatsCollector! + Collected stats for 97889 documents + Created dictionary for INT column: homeRuns with cardinality: 67, range: 0 to 73 + Created dictionary for INT column: playerStint with cardinality: 5, range: 1 to 5 + Created dictionary for INT column: groundedIntoDoublePlays with cardinality: 35, range: 0 to 36 + Created dictionary for INT column: numberOfGames with cardinality: 165, range: 1
[incubator-pinot] branch add-docs deleted (was 9b50d89)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch add-docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 9b50d89 Add documentation The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Spelling correction (#3977)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 581464e Spelling correction (#3977) 581464e is described below commit 581464ee63529b53ba528f84e970f50e0e2e7d5e Author: Shrinivas Devshatwar <47772040+sdevshat...@users.noreply.github.com> AuthorDate: Mon Mar 18 06:47:53 2019 -0700 Spelling correction (#3977) --- docs/client_api.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/client_api.rst b/docs/client_api.rst index e9c1a16..04e9bf0 100644 --- a/docs/client_api.rst +++ b/docs/client_api.rst @@ -20,7 +20,7 @@ Executing queries via REST API on the Broker -The Pinot REST API can be accessed by invoking ``POST`` operation witha a JSON body containing the parameter ``pql`` +The Pinot REST API can be accessed by invoking ``POST`` operation with a JSON body containing the parameter ``pql`` to the ``/query`` URI endpoint on a broker. Depending on the type of query, the results can take different shapes. The examples below use curl. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Make RetentionManager and OfflineSegmentIntervalChecker initial delays configurable (#3946)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new b2c5a88 Make RetentionManager and OfflineSegmentIntervalChecker
initial delays configurable (#3946)
b2c5a88 is described below
commit b2c5a882fb35f1be7bcaeb0f6c65c28a8f1d7ff9
Author: Sunitha Beeram
AuthorDate: Fri Mar 15 11:50:17 2019 -0700
Make RetentionManager and OfflineSegmentIntervalChecker initial delays
configurable (#3946)
---
.../java/org/apache/pinot/controller/ControllerConf.java | 14 ++
.../controller/helix/core/retention/RetentionManager.java | 2 +-
.../validation/OfflineSegmentIntervalChecker.java | 2 +-
3 files changed, 16 insertions(+), 2 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
index 05203ae..1179996 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
@@ -84,6 +84,10 @@ public class ControllerConf extends PropertiesConfiguration {
// Initial delays
private static final String STATUS_CHECKER_INITIAL_DELAY_IN_SECONDS =
"controller.statusChecker.initialDelayInSeconds";
+private static final String RETENTION_MANAGER_INITIAL_DELAY_IN_SECONDS =
+"controller.retentionManager.initialDelayInSeconds";
+private static final String
OFFLINE_SEGMENT_INTERVAL_CHECKER_INITIAL_DELAY_IN_SECONDS =
+"controller.offlineSegmentIntervalChecker.initialDelayInSeconds";
public static final int MIN_INITIAL_DELAY_IN_SECONDS = 120;
public static final int MAX_INITIAL_DELAY_IN_SECONDS = 300;
@@ -575,6 +579,16 @@ public class ControllerConf extends
PropertiesConfiguration {
ControllerPeriodicTasksConf.getRandomInitialDelayInSeconds());
}
+ public long getRetentionManagerInitialDelayInSeconds() {
+return
getLong(ControllerPeriodicTasksConf.RETENTION_MANAGER_INITIAL_DELAY_IN_SECONDS,
+ControllerPeriodicTasksConf.getRandomInitialDelayInSeconds());
+ }
+
+ public long getOfflineSegmentIntervalCheckerInitialDelayInSeconds() {
+return
getLong(ControllerPeriodicTasksConf.OFFLINE_SEGMENT_INTERVAL_CHECKER_INITIAL_DELAY_IN_SECONDS,
+ControllerPeriodicTasksConf.getRandomInitialDelayInSeconds());
+ }
+
public void setStatusCheckerInitialDelayInSeconds(long
initialDelayInSeconds) {
setProperty(ControllerPeriodicTasksConf.STATUS_CHECKER_INITIAL_DELAY_IN_SECONDS,
initialDelayInSeconds);
}
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
index b3abf07..b9f2bd5e 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/retention/RetentionManager.java
@@ -57,7 +57,7 @@ public class RetentionManager extends
ControllerPeriodicTask {
public RetentionManager(PinotHelixResourceManager pinotHelixResourceManager,
ControllerConf config,
ControllerMetrics controllerMetrics) {
super("RetentionManager",
config.getRetentionControllerFrequencyInSeconds(),
-config.getPeriodicTaskInitialDelayInSeconds(),
pinotHelixResourceManager, controllerMetrics);
+config.getRetentionManagerInitialDelayInSeconds(),
pinotHelixResourceManager, controllerMetrics);
_deletedSegmentsRetentionInDays =
config.getDeletedSegmentsRetentionInDays();
LOGGER.info("Starting RetentionManager with runFrequencyInSeconds: {},
deletedSegmentsRetentionInDays: {}",
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/validation/OfflineSegmentIntervalChecker.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/validation/OfflineSegmentIntervalChecker.java
index 2646ddd..b01aa27 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/validation/OfflineSegmentIntervalChecker.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/validation/OfflineSegmentIntervalChecker.java
@@ -51,7 +51,7 @@ public class OfflineSegmentIntervalChecker extends
ControllerPeriodicTask
public OfflineSegmentIntervalChecker(ControllerConf config,
PinotHelixResourceManager pinotHelixResourceManager,
ValidationMetrics validationMetrics, ControllerMetrics
controllerMetrics) {
super("OfflineSegmentIntervalChecker",
config.getOfflineSegmentIntervalCheckerFrequencyInSeconds(),
-config.getPeriodicTaskInitialDel
[incubator-pinot] branch tuning-scatter-gather-doc deleted (was 04073b7)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch tuning-scatter-gather-doc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 04073b7 Add documentation for tuning scatter and gather The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Add more documents in Admin Guide (#3972)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 9ded2a6 Add more documents in Admin Guide (#3972)
9ded2a6 is described below
commit 9ded2a644485d45041a2470d20d04b460d5e507b
Author: Jialiang Li
AuthorDate: Thu Mar 14 12:16:40 2019 -0700
Add more documents in Admin Guide (#3972)
---
docs/img/generate-segment.png | Bin 0 -> 218597 bytes
docs/img/list-schemas.png | Bin 0 -> 8952 bytes
docs/img/query-table.png | Bin 0 -> 35914 bytes
docs/img/upload-segment.png | Bin 0 -> 13944 bytes
docs/management_api.rst | 32 ---
docs/pinot_hadoop.rst | 71 --
6 files changed, 96 insertions(+), 7 deletions(-)
diff --git a/docs/img/generate-segment.png b/docs/img/generate-segment.png
new file mode 100644
index 000..5848781
Binary files /dev/null and b/docs/img/generate-segment.png differ
diff --git a/docs/img/list-schemas.png b/docs/img/list-schemas.png
new file mode 100644
index 000..9b00855
Binary files /dev/null and b/docs/img/list-schemas.png differ
diff --git a/docs/img/query-table.png b/docs/img/query-table.png
new file mode 100644
index 000..5859f2b
Binary files /dev/null and b/docs/img/query-table.png differ
diff --git a/docs/img/upload-segment.png b/docs/img/upload-segment.png
new file mode 100644
index 000..edd348d
Binary files /dev/null and b/docs/img/upload-segment.png differ
diff --git a/docs/management_api.rst b/docs/management_api.rst
index f8483eb..5a7a0e4 100644
--- a/docs/management_api.rst
+++ b/docs/management_api.rst
@@ -17,11 +17,35 @@
.. under the License.
..
-Managing Pinot via REST API on the Controller
-=
+Managing Pinot
+==
-*TODO* : Remove this section altogether and find a place somewhere for a
pointer to the management API. Maybe in the 'Running pinot in production'
section?
+There are two ways to manage Pinot cluster, i.e. using Pinot management
console and ``pinot-admin.sh`` script.
-There is a REST API which allows management of tables, tenants, segments and
schemas. It can be accessed by going to ``http://[controller host]/help`` which
offers a web UI to do these tasks, as well as document the REST API.
+Pinot Management Console
+
+
+There is a REST API which allows management of tables, tenants, segments and
schemas. It can be accessed by going to
+``http://[controller host]/help`` which offers a web UI to do these tasks, as
well as document the REST API.
+
+For example, list all the schema within Pinot cluster:
+
+ .. figure:: img/list-schemas.png
+
+Upload a pinot segment:
+
+ .. figure:: img/upload-segment.png
+
+
+Pinot-admin.sh
+--
It can be used instead of the ``pinot-admin.sh`` commands to automate the
creation of tables and tenants.
+
+For example, create a pinot segment:
+
+ .. figure:: img/generate-segment.png
+
+Query a table:
+
+ .. figure:: img/query-table.png
diff --git a/docs/pinot_hadoop.rst b/docs/pinot_hadoop.rst
index 1611040..95918d2 100644
--- a/docs/pinot_hadoop.rst
+++ b/docs/pinot_hadoop.rst
@@ -23,7 +23,8 @@ Creating Pinot segments
===
Pinot segments can be created offline on Hadoop, or via command line from data
files. Controller REST endpoint
-can then be used to add the segment to the table to which the segment belongs.
+can then be used to add the segment to the table to which the segment belongs.
Pinot segments can also be created by
+ingesting data from realtime resources (such as Kafka).
Creating segments using hadoop
--
@@ -177,8 +178,8 @@ Sample Schema:
"schemaName" : "mySchema",
}
-Pushing segments to Pinot
-^
+Pushing offline segments to Pinot
+^
You can use curl to push a segment to pinot:
@@ -196,3 +197,67 @@ Alternatively you can use the pinot-admin.sh utility to
upload one or more segme
The command uploads all the segments found in ``segmentDirectoryPath``.
The segments could be either tar-compressed (in which case it is a file under
``segmentDirectoryPath``)
or uncompressed (in which case it is a directory under
``segmentDirectoryPath``).
+
+Realtime segment generation
+^^^
+
+To consume in realtime, we simply need to create a table that uses the same
schema and points to the Kafka topic to
+consume from, using a table definition such as this one:
+
+.. code-block:: none
+ {
+"tableName":"flights",
+"segmentsConfig" : {
+"retentionTimeUnit":"DAYS",
+"retentionTimeValue":"7",
+"segmentPushFrequency&qu
[incubator-pinot] branch master updated: Update dev guide to include toc for high level sections (#3973)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 2411d89 Update dev guide to include toc for high level sections (#3973) 2411d89 is described below commit 2411d89cc699749fa6edadce3850ccfb0f758581 Author: Sunitha Beeram AuthorDate: Thu Mar 14 11:49:12 2019 -0700 Update dev guide to include toc for high level sections (#3973) --- docs/code_modules.rst| 6 +- docs/contribution_guidelines.rst | 19 ++- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/docs/code_modules.rst b/docs/code_modules.rst index 6a01abf..33e5a20 100644 --- a/docs/code_modules.rst +++ b/docs/code_modules.rst @@ -19,8 +19,12 @@ .. _code-modules: + +* Code Modules and Organization -= +* + +.. contents:: Table of Contents Before proceeding to contributing changes to Pinot, review the contents of this section. diff --git a/docs/contribution_guidelines.rst b/docs/contribution_guidelines.rst index d2e7a81..e980158 100644 --- a/docs/contribution_guidelines.rst +++ b/docs/contribution_guidelines.rst @@ -17,21 +17,24 @@ .. under the License. .. +*** Contribution Guidelines -=== +*** + +.. contents:: Table of Contents Before you begin to contribute, make sure you have reviewed :ref:`dev-setup` and :ref:`code-modules` sections and that you have created your own fork of the pinot source code. Create an issue for the change -^^ +-- Create a Pinot issue for the change you would like to make. Provide information on why the change is needed and how you plan to address it. Use the conversations on the issue as a way to validate assumptions and the right way to proceed. -Be sure to review sections on :ref:`compatiblity-changes` and :ref:`external-libs`. +Be sure to review sections on :ref:`compatibility-changes` and :ref:`external-libs`. Once you are clear about what you want to do, proceed with the next steps listed below. Create a branch for your change -^^^ +--- .. code-block:: none @@ -48,6 +51,11 @@ Create a branch for your change Make the necessary changes. If the changes you plan to make are too big, make sure you break it down into smaller tasks. +Making the changes +-- + +Follow the recommendations/best-practices noted here when you are making changes. + Code documentation ^^ Please ensure your code is adequately documented. Some things to consider for documentation: @@ -152,7 +160,7 @@ If attention to the licensing terms in not paid early on, they will be caught mu Updating code at that time to work with the right libraries at that time might require bigger refactoring changes and delay the release process. Creating a PR -^ +- * Verifying code-style @@ -214,6 +222,7 @@ At this time, if rebase flags any conflicts, resolve the conflicts and follow th Run additional tests/validations for the new changes and update the PR by pushing your changes: .. code-block:: none + $ git push origin * Once your change is merged, check to see if any documentation needs to be updated. If so, create a PR for documentation. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Improving docs on index techniques under tuning pinot section. (#3964)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 9e37800 Improving docs on index techniques under tuning pinot
section. (#3964)
9e37800 is described below
commit 9e37800f2b2d33ba173e4bdf0ca03e1a7a126bf4
Author: Seunghyun Lee
AuthorDate: Wed Mar 13 08:35:48 2019 -0700
Improving docs on index techniques under tuning pinot section. (#3964)
---
docs/conf.py | 4 +-
docs/img/dictionary.png | Bin 0 -> 80599 bytes
docs/img/no-dictionary.png | Bin 0 -> 111237 bytes
docs/img/sorted-forward.png | Bin 0 -> 70022 bytes
docs/img/sorted-inverted.png | Bin 0 -> 89473 bytes
docs/index_techniques.rst| 109 +++
6 files changed, 102 insertions(+), 11 deletions(-)
diff --git a/docs/conf.py b/docs/conf.py
index e34396a..667bd8f 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -76,9 +76,9 @@ author = u'Pinot development team'
# built documents.
#
# The short X.Y version.
-version = u'0.016'
+version = u'0.1.0'
# The full version, including alpha/beta/rc tags.
-release = u'0.016'
+release = u'0.1.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
diff --git a/docs/img/dictionary.png b/docs/img/dictionary.png
new file mode 100644
index 000..d4c3d20
Binary files /dev/null and b/docs/img/dictionary.png differ
diff --git a/docs/img/no-dictionary.png b/docs/img/no-dictionary.png
new file mode 100644
index 000..33877f7
Binary files /dev/null and b/docs/img/no-dictionary.png differ
diff --git a/docs/img/sorted-forward.png b/docs/img/sorted-forward.png
new file mode 100644
index 000..9c2e9a6
Binary files /dev/null and b/docs/img/sorted-forward.png differ
diff --git a/docs/img/sorted-inverted.png b/docs/img/sorted-inverted.png
new file mode 100644
index 000..9d607a6
Binary files /dev/null and b/docs/img/sorted-inverted.png differ
diff --git a/docs/index_techniques.rst b/docs/index_techniques.rst
index 51f0d1a..a3ade2a 100644
--- a/docs/index_techniques.rst
+++ b/docs/index_techniques.rst
@@ -24,19 +24,29 @@ Index Techniques
Pinot currently supports the following index techniques, where each of them
have their own advantages in different query
-scenarios.
+scenarios. By default, Pinot will use ``dictionary-encoded forward index`` for
each column.
Forward Index
-
-Dictionary-Encoded Forward Index with Bit Compression
-~
+Dictionary-Encoded Forward Index with Bit Compression (Default)
+~~~
For each unique value from a column, we assign an id to it, and build a
dictionary from the id to the value. Then in the
forward index, we only store the bit-compressed ids instead of the values.
With few number of unique values, dictionary-encoding can significantly
improve the space efficiency of the storage.
+The below diagram shows the dictionary encoding for two columns with
``integer`` and ``string`` types. As seen in the
+``colA``, dictionary encoding will save significant amount of space for
duplicated values. On the other hand, ``colB``
+has no duplicated data. Dictionary encoding will not compress much data in
this case where there are a lot of unique
+values in the column. For ``string`` type, we pick the length of the longest
value and use it as the length for
+dictionary's fixed length value array. In this case, padding overhead can be
high if there are a large number of unique
+values for a column.
+
+.. image:: img/dictionary.png
+
+
Raw Value Forward Index
~~~
@@ -45,13 +55,61 @@ In contrast to the dictionary-encoded forward index, raw
value forward index dir
Without the dictionary, the dictionary lookup step can be skipped for each
value fetch. Also, the index can take
advantage of the good locality of the values, thus improve the performance of
scanning large number of values.
+A typical use case to apply raw value forward index is when the column has a
large number of unique values and the
+dictionary does not provide much compression. As seen the above diagram for
dictionary encoding, scanning values
+with a dictionary involves a lot of random access because we need to perform
dictionary look up. On the other hand,
+we can scan values sequentially with raw value forward index and this can
improve performance a lot when applied
+appropriately.
+
+.. image:: img/no-dictionary.png
+
+Raw value forward index can be configured for a table by setting it in the
table config as
+
+.. code-block:: none
+
+{
+"noDictionaryColumns": [
+ "column_name",
+ ...
+],
+...
+}
+
[incubator-pinot] branch improve-index-techniques deleted (was 3a26d96)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch improve-index-techniques in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 3a26d96 Improving docs on index techniques under tuning pinot section. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: readthedocs: Update instructions for using fork (#3937)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new bf161ec readthedocs: Update instructions for using fork (#3937) bf161ec is described below commit bf161ecb0ec59c1f967bf47e45b8ed9c6316264d Author: Sunitha Beeram AuthorDate: Mon Mar 11 13:45:52 2019 -0700 readthedocs: Update instructions for using fork (#3937) * Update instructions for using fork * More info on fork * Update guidelines for working with forks * Address review comments --- docs/contribution_guidelines.rst | 22 +++--- docs/dev_env.rst | 13 + 2 files changed, 24 insertions(+), 11 deletions(-) diff --git a/docs/contribution_guidelines.rst b/docs/contribution_guidelines.rst index b1620c1..d2e7a81 100644 --- a/docs/contribution_guidelines.rst +++ b/docs/contribution_guidelines.rst @@ -20,7 +20,7 @@ Contribution Guidelines === -Before you begin to contribute, make sure you have reviewed :ref:`dev-setup` and :ref:`code-modules` sections. +Before you begin to contribute, make sure you have reviewed :ref:`dev-setup` and :ref:`code-modules` sections and that you have created your own fork of the pinot source code. Create an issue for the change ^^ @@ -35,7 +35,15 @@ Create a branch for your change .. code-block:: none - $ cd incubator-pinot + $ cd pinot + # + # ensure you are starting from the latest code base + # the following steps, ensure your fork's (origin's) master is up-to-date + # + $ git fetch upstream + $ git checkout master + $ git merge upstream/master + # create a branch for your issue $ git checkout -b Make the necessary changes. If the changes you plan to make are too big, make sure you break it down into smaller tasks. @@ -117,7 +125,7 @@ Identify a list of tests for the changes you have made. Depending on the scope o License Headers for newly added files ^ -All source code files should have license headers. To automatically add the header for any new file you plan to checkin, run in ``incubator-pinot`` top-level folder: +All source code files should have license headers. To automatically add the header for any new file you plan to checkin, run in ``pinot`` top-level folder: .. code-block:: none @@ -136,9 +144,9 @@ If you checkin third-party code or files, please make sure you review Apache gui Once you determine the code you are pulling in adhere to the guidelines above, go ahead pull the changes in. Do not add license headers for them. Follow these instructions to ensure we are compliant with Apache Licensing process: -* Under ``incubator-pinot/licenses`` add a LICENSE- file that has the license terms of the included library. -* Update the ``incubator-pinot/LICENSE`` file to indicate the newly added library file paths under the corresponding supported Licenses. -* Update the exclusion rules for ``license`` and ``rat`` maven plugins in the parent pom: ``incubator-pinot/pom.xml``. +* Under ``pinot/licenses`` add a LICENSE- file that has the license terms of the included library. +* Update the ``pinot/LICENSE`` file to indicate the newly added library file paths under the corresponding supported Licenses. +* Update the exclusion rules for ``license`` and ``rat`` maven plugins in the parent pom: ``pinot/pom.xml``. If attention to the licensing terms in not paid early on, they will be caught much later in the process, when we prepare to make a new release. Updating code at that time to work with the right libraries at that time might require bigger refactoring changes and delay the release process. @@ -197,7 +205,7 @@ It is possible that while your change is being reviewed, other changes were made $ git commit -m "Meaningful message for the udpate" # pull new changes $ git checkout master - $ git pull + $ git merge upstream/master $ git checkout $ git rebase master diff --git a/docs/dev_env.rst b/docs/dev_env.rst index dd8aee6..f7a7fb7 100644 --- a/docs/dev_env.rst +++ b/docs/dev_env.rst @@ -37,16 +37,21 @@ Getting the Source Code Create a fork ^ To limit the number of branches created on the Apache Pinot repository, we recommend that you create a fork by clicking on the fork button `here <https://github.com/apache/incubator-pinot>`_. +Read more about `fork workflow here <https://www.atlassian.com/git/tutorials/comparing-workflows/forking-workflow>`_ Clone the repository locally .. code-block:: none - $mkdir workspace - $cd workspace - $git clone https://github.com//incubator-pinot.git - $cd incubator-pinot + $ mkdir workspace + $ cd workspace + $ git clone g...@github.com:/pinot.git
[incubator-pinot] branch master updated: Provide links for high level sections (#3939)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 0c3519b Provide links for high level sections (#3939)
0c3519b is described below
commit 0c3519b86d948d0fb8043cebec34a2907ed326ee
Author: Sunitha Beeram
AuthorDate: Fri Mar 8 13:26:40 2019 -0800
Provide links for high level sections (#3939)
---
docs/admin_guide.rst | 6 ++
docs/dev_guide.rst| 6 ++
docs/index.rst| 20 +---
docs/{user_guide.rst => introduction.rst} | 14 ++
docs/user_guide.rst | 6 ++
5 files changed, 33 insertions(+), 19 deletions(-)
diff --git a/docs/admin_guide.rst b/docs/admin_guide.rst
index 422f56a..51c5ec0 100644
--- a/docs/admin_guide.rst
+++ b/docs/admin_guide.rst
@@ -17,6 +17,12 @@
.. under the License.
..
+###
+Admin Guide
+###
+
+.. _admin-guide:
+
.. toctree::
:maxdepth: 1
diff --git a/docs/dev_guide.rst b/docs/dev_guide.rst
index 1d1a05a..90df29d 100644
--- a/docs/dev_guide.rst
+++ b/docs/dev_guide.rst
@@ -17,6 +17,12 @@
.. under the License.
..
+###
+Developer Guide
+###
+
+.. _dev-guide:
+
.. toctree::
:maxdepth: 1
diff --git a/docs/index.rst b/docs/index.rst
index 4a4d5b0..0dabf7f 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -26,32 +26,22 @@
Pinot
#
-.. _introduction:
.. toctree::
- :caption: Introduction
- :maxdepth: 1
+ :maxdepth: 2
- intro
- architecture
- getting_started
+ introduction
-.. _user-guide:
.. toctree::
- :caption: User Guide
- :maxdepth: 1
+ :maxdepth: 2
user_guide
-.. _admin_guide:
.. toctree::
- :caption: Admin Guide
- :maxdepth: 1
+ :maxdepth: 2
admin_guide
-.. _dev_guide:
.. toctree::
- :caption: Developer Guide
- :maxdepth: 1
+ :maxdepth: 2
dev_guide
\ No newline at end of file
diff --git a/docs/user_guide.rst b/docs/introduction.rst
similarity index 87%
copy from docs/user_guide.rst
copy to docs/introduction.rst
index 49babcc..384df02 100644
--- a/docs/user_guide.rst
+++ b/docs/introduction.rst
@@ -17,9 +17,15 @@
.. under the License.
..
-.. toctree::
- :maxdepth: 1
+
+Introduction
+
+
+.. _introduction:
- pql_examples
- client_api
+.. toctree::
+:maxdepth: 1
+intro
+architecture
+getting_started
diff --git a/docs/user_guide.rst b/docs/user_guide.rst
index 49babcc..7303d3c 100644
--- a/docs/user_guide.rst
+++ b/docs/user_guide.rst
@@ -17,6 +17,12 @@
.. under the License.
..
+##
+User Guide
+##
+
+.. _user-guide:
+
.. toctree::
:maxdepth: 1
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: readthedocs: Add contribution guidelines (#3925)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 63f0d3d readthedocs: Add contribution guidelines (#3925) 63f0d3d is described below commit 63f0d3d386079b0ba63a880eaf84464742ba4814 Author: Sunitha Beeram AuthorDate: Thu Mar 7 12:00:21 2019 -0800 readthedocs: Add contribution guidelines (#3925) * Add contribution guidelines * Add info about travis * Address review comments --- docs/code_modules.rst| 19 - docs/contribution_guidelines.rst | 167 --- 2 files changed, 173 insertions(+), 13 deletions(-) diff --git a/docs/code_modules.rst b/docs/code_modules.rst index adb1fc7..6a01abf 100644 --- a/docs/code_modules.rst +++ b/docs/code_modules.rst @@ -59,6 +59,10 @@ overall executable size for individually deployable component. Each module has a ``src/main/java`` folder where the code resides and ``src/test/java`` where the *unit* tests corresponding to the module's code reside. +.. _pinot-foundation: + +Foundational modules + The following figure provides a high-level overview of the foundational Pinot modules. .. figure:: img/PinotFoundation.png @@ -132,10 +136,21 @@ Auxiliary modules - In addition to the core modules described above, Pinot code provides the following modules: -* ``pinot-tools``: This module is a collection of many tools useful for setting up Pinot cluster, creating/updating segments. It also houses the Pinot quick start guide code. +* ``pinot-tools``: This module is a collection of many tools useful for setting up Pinot cluster, creating/updating segments. +It also houses the Pinot quick start guide code. * ``pinot-perf``: This module has a collection of benchmark test code used to evaluate design options. * ``pinot-client-api``: This module houses the Java client API. See :ref:`java-client` for more info. -* ``pinot-integration-tests``: This module holds integration tests that test functionality across multiple classes or components. These tests typically do not rely on mocking and provide more end to end coverage for code. +* ``pinot-integration-tests``: This module holds integration tests that test functionality across multiple classes or components. +These tests typically do not rely on mocking and provide more end to end coverage for code. + +.. _extension-modules: + +Extension modules +- +``pinot-hadoop-filesystem`` and ``pinot-azure-filesystem`` are module added to support extensions to Pinot filesystem. +The functionality is broken down into modules of their own to avoid polluting the common modules with additional large libraries. +These libraries bring in transitive dependencies of their own that can cause classpath conflicts at runtime. We would like to +avoid this for the common usage of Pinot as much as possible. \ No newline at end of file diff --git a/docs/contribution_guidelines.rst b/docs/contribution_guidelines.rst index 0ac2f86..b1620c1 100644 --- a/docs/contribution_guidelines.rst +++ b/docs/contribution_guidelines.rst @@ -26,41 +26,186 @@ Create an issue for the change ^^ Create a Pinot issue for the change you would like to make. Provide information on why the change is needed and how you plan to address it. Use the conversations on the issue as a way to validate assumptions and the right way to proceed. +Be sure to review sections on :ref:`compatiblity-changes` and :ref:`external-libs`. -Once you have a good about what you want to do, proceed with the next steps. +Once you are clear about what you want to do, proceed with the next steps listed below. Create a branch for your change ^^^ +.. code-block:: none + + $ cd incubator-pinot + $ git checkout -b + +Make the necessary changes. If the changes you plan to make are too big, make sure you break it down into smaller tasks. + +Code documentation +^^ +Please ensure your code is adequately documented. Some things to consider for documentation: + +* Always include class level java docs. +At the top class level, we are looking for information about what functionality is provided by the class, +what state is maintained by the class, whether there are concurrency/thread-safety concerns and any exceptional behavior that the class might exhibit. + +* Document public methods and their parameters. + +Logging +^^^ + +* Ensure there is adequate logging for positive paths as well as exceptional paths. As a corollary to this, ensure logs are not noisy. +* Do not use System.out.println to log messages. Use the ``slf4j`` loggers. +* Use logging levels correctly: set level to ``debug`` for verbose logs useful for only for debugging. +* Do not log
[incubator-pinot] branch contrib deleted (was 923e486)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch contrib in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 923e486 Address review comments The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch contrib updated: Address review comments
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch contrib in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/contrib by this push: new 923e486 Address review comments 923e486 is described below commit 923e486301c5f0f7200993444736aadd035935d8 Author: Sunitha Beeram AuthorDate: Thu Mar 7 11:31:08 2019 -0800 Address review comments --- docs/contribution_guidelines.rst | 31 +++ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/docs/contribution_guidelines.rst b/docs/contribution_guidelines.rst index 5fd2fe1..b1620c1 100644 --- a/docs/contribution_guidelines.rst +++ b/docs/contribution_guidelines.rst @@ -35,8 +35,8 @@ Create a branch for your change .. code-block:: none - $cd incubator-pinot - $git checkout -b + $ cd incubator-pinot + $ git checkout -b Make the necessary changes. If the changes you plan to make are too big, make sure you break it down into smaller tasks. @@ -117,8 +117,7 @@ Identify a list of tests for the changes you have made. Depending on the scope o License Headers for newly added files ^ -All source code files should have license headers. To automatically add the header for any new file you plan to checkin, - run: +All source code files should have license headers. To automatically add the header for any new file you plan to checkin, run in ``incubator-pinot`` top-level folder: .. code-block:: none @@ -126,7 +125,15 @@ All source code files should have license headers. To automatically add the head .. note:: -If you checkin third-party code or files, ensure they are distributed under Apache, MIT or BSD-3 licenses. +If you checkin third-party code or files, please make sure you review Apache guidelines: + +* `Licences that can be included <https://www.apache.org/legal/resolved.html#what-can-we-include-in-an-asf-project-category-a>`_ + +* `Licences that may be included <https://www.apache.org/legal/resolved.html#what-can-we-maybe-include-in-an-asf-project-category-b>`_ + +* `Licenses that should not be included <https://www.apache.org/legal/resolved.html#what-can-we-not-include-in-an-asf-project-category-x>`_ + +Once you determine the code you are pulling in adhere to the guidelines above, go ahead pull the changes in. Do not add license headers for them. Follow these instructions to ensure we are compliant with Apache Licensing process: * Under ``incubator-pinot/licenses`` add a LICENSE- file that has the license terms of the included library. @@ -186,19 +193,19 @@ It is possible that while your change is being reviewed, other changes were made .. code-block:: none # commit your changes - $git add - $git commit -m "Meaningful message for the udpate" + $ git add + $ git commit -m "Meaningful message for the udpate" # pull new changes - $git checkout master - $git pull - $git checkout - $git rebase master + $ git checkout master + $ git pull + $ git checkout + $ git rebase master At this time, if rebase flags any conflicts, resolve the conflicts and follow the instructions provided by the rebase command. Run additional tests/validations for the new changes and update the PR by pushing your changes: .. code-block:: none - $git push origin + $ git push origin * Once your change is merged, check to see if any documentation needs to be updated. If so, create a PR for documentation. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Fix formatting issues and figures used (#3923)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new e6dfb05 Fix formatting issues and figures used (#3923) e6dfb05 is described below commit e6dfb050c0a1118bf54dc493fe9efd863f416765 Author: Sunitha Beeram AuthorDate: Thu Mar 7 11:23:19 2019 -0800 Fix formatting issues and figures used (#3923) --- docs/README.md | 43 --- docs/dev_env.rst | 15 ++- docs/img/PinotBroker.png | Bin 31737 -> 40314 bytes docs/img/PinotController.png | Bin 43621 -> 47991 bytes docs/img/PinotFoundation.png | Bin 35783 -> 44122 bytes docs/img/PinotServer.png | Bin 43540 -> 55062 bytes 6 files changed, 34 insertions(+), 24 deletions(-) diff --git a/docs/README.md b/docs/README.md index 0e41afc..5f23fcc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,23 +1,28 @@ -.. -.. Licensed to the Apache Software Foundation (ASF) under one -.. or more contributor license agreements. See the NOTICE file -.. distributed with this work for additional information -.. regarding copyright ownership. The ASF licenses this file -.. to you under the Apache License, Version 2.0 (the -.. "License"); you may not use this file except in compliance -.. with the License. You may obtain a copy of the License at -.. -.. http://www.apache.org/licenses/LICENSE-2.0 -.. -.. Unless required by applicable law or agreed to in writing, -.. software distributed under the License is distributed on an -.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -.. KIND, either express or implied. See the License for the -.. specific language governing permissions and limitations -.. under the License. -.. + Updating the docs: 1. Edit or add files as needed. 2. Run "make html" -3. Open _build/html/index.html in your favorite browser and ensure contents and links work correctly. \ No newline at end of file +3. Open _build/html/index.html in your favorite browser and ensure contents and links work correctly. + +NOTE: +You may see some differences locally as the version of sphinx-build on your local host might not be the same as the one used in readthedocs.io for building the pinot website docs. \ No newline at end of file diff --git a/docs/dev_env.rst b/docs/dev_env.rst index d54bb45..dd8aee6 100644 --- a/docs/dev_env.rst +++ b/docs/dev_env.rst @@ -28,17 +28,21 @@ Dev Environment Setup To contribute to Pinot, please follow the instructions below. +Git +--- +Pinot uses git for source code management. If you are new to Git, it will be good to review `basics <https://git-scm.com/book/en/v1/Getting-Started-Git-Basics>`_ of Git and a common tasks like `managing branches <https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell>`_ and `rebasing <https://git-scm.com/book/en/v2/Git-Branching-Rebasing>`_. + Getting the Source Code --- - Create a fork ^ -To limit the number of branches created on the main Pinot repository, we recommend that you create a fork by clicking on the fork button `here <https://github.com/apache/incubator-pinot>`_. - -.. code-block:: none +To limit the number of branches created on the Apache Pinot repository, we recommend that you create a fork by clicking on the fork button `here <https://github.com/apache/incubator-pinot>`_. Clone the repository locally + +.. code-block:: none + $mkdir workspace $cd workspace $git clone https://github.com//incubator-pinot.git @@ -47,9 +51,10 @@ Clone the repository locally Maven - Pinot is a Maven project and familiarity with Maven will help you work with Pinot code. If you are new to Maven, you can -more `here `_ and `here <http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html>`_. +read about Maven `here `_ and `get a quick overview here <http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html>`_. Run the following maven command to setup the project. + .. code-block:: none # compile, download sources diff --git a/docs/img/PinotBroker.png b/docs/img/PinotBroker.png index 675834d..f14badd 100644 Binary files a/docs/img/PinotBroker.png and b/docs/img/PinotBroker.png differ diff --git a/docs/img/PinotController.png b/docs/img/PinotController.png index 451e20b..63a658b 100644 Binary files a/docs/img/PinotController.png and b/docs/img/PinotController.png differ diff --git a/docs/img/PinotFoundation.png b/docs/img/PinotFoundation.png index 889f2fc..91a85a4 100644 Binary files a/docs/img/PinotFoundation.png and b/docs/img/PinotFoundation.png differ diff --git a/docs/img/PinotServer.png b/docs/img/PinotServer.png index f2789bc..38ac17e
[incubator-pinot] branch docs deleted (was 068fc52)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 068fc52 Fix formatting issues and figures used The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch contrib updated (281d949 -> a5038c6)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch contrib in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 281d949 Add contribution guidelines new a5038c6 Add contribution guidelines This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (281d949) \ N -- N -- N refs/heads/contrib (a5038c6) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 5803 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/contribution_guidelines.rst | 15 +++ 1 file changed, 11 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Add contribution guidelines
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch contrib in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 281d949789df8cfc949b4881178cb5b66dc47811 Author: Sunitha Beeram AuthorDate: Thu Mar 7 08:27:22 2019 -0800 Add contribution guidelines --- docs/code_modules.rst| 19 - docs/contribution_guidelines.rst | 148 --- 2 files changed, 155 insertions(+), 12 deletions(-) diff --git a/docs/code_modules.rst b/docs/code_modules.rst index adb1fc7..6a01abf 100644 --- a/docs/code_modules.rst +++ b/docs/code_modules.rst @@ -59,6 +59,10 @@ overall executable size for individually deployable component. Each module has a ``src/main/java`` folder where the code resides and ``src/test/java`` where the *unit* tests corresponding to the module's code reside. +.. _pinot-foundation: + +Foundational modules + The following figure provides a high-level overview of the foundational Pinot modules. .. figure:: img/PinotFoundation.png @@ -132,10 +136,21 @@ Auxiliary modules - In addition to the core modules described above, Pinot code provides the following modules: -* ``pinot-tools``: This module is a collection of many tools useful for setting up Pinot cluster, creating/updating segments. It also houses the Pinot quick start guide code. +* ``pinot-tools``: This module is a collection of many tools useful for setting up Pinot cluster, creating/updating segments. +It also houses the Pinot quick start guide code. * ``pinot-perf``: This module has a collection of benchmark test code used to evaluate design options. * ``pinot-client-api``: This module houses the Java client API. See :ref:`java-client` for more info. -* ``pinot-integration-tests``: This module holds integration tests that test functionality across multiple classes or components. These tests typically do not rely on mocking and provide more end to end coverage for code. +* ``pinot-integration-tests``: This module holds integration tests that test functionality across multiple classes or components. +These tests typically do not rely on mocking and provide more end to end coverage for code. + +.. _extension-modules: + +Extension modules +- +``pinot-hadoop-filesystem`` and ``pinot-azure-filesystem`` are module added to support extensions to Pinot filesystem. +The functionality is broken down into modules of their own to avoid polluting the common modules with additional large libraries. +These libraries bring in transitive dependencies of their own that can cause classpath conflicts at runtime. We would like to +avoid this for the common usage of Pinot as much as possible. \ No newline at end of file diff --git a/docs/contribution_guidelines.rst b/docs/contribution_guidelines.rst index 0ac2f86..173e488 100644 --- a/docs/contribution_guidelines.rst +++ b/docs/contribution_guidelines.rst @@ -26,25 +26,93 @@ Create an issue for the change ^^ Create a Pinot issue for the change you would like to make. Provide information on why the change is needed and how you plan to address it. Use the conversations on the issue as a way to validate assumptions and the right way to proceed. +Be sure to review sections on :ref:`compatiblity-changes` and :ref:`external-libs`. -Once you have a good about what you want to do, proceed with the next steps. +Once you are clear about what you want to do, proceed with the next steps listed below. Create a branch for your change ^^^ -Creating a PR -^ +.. code-block:: none -* Verifying code-style + $cd incubator-pinot + $git checkout -b +Make the necessary changes. If the changes you plan to make are too big, make sure you break it down into smaller tasks. -Run the following command to verify the code-style before posting a PR +Code documentation +^^ +Please ensure your code is adequately documented. Some things to consider for documentation: -.. code-block:: none +* Always include class level java docs. +At the top class level, we are looking for information about what functionality is provided by the class, +what state is maintained by the class, whether there are concurrency/thread-safety concerns and any exceptional behavior that the class might exhibit. -mvn checkstyle:check +* Document public methods and their parameters. + +Logging +^^^ + +* Ensure there is adequate logging for positive paths as well as exceptional paths. As a corollary to this, ensure logs are not noisy. +* Do not use System.out.println to log messages. Use the ``slf4j`` loggers. +* Use logging levels correctly: set level to ``debug`` for verbose logs useful for only for debugging. +* Do not log stack traces via ``printStackTrace`` method of the exception. + +Exceptions and Exception-Handling
[incubator-pinot] 01/01: Fix formatting issues and figures used
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 068fc52e2ccae84f4abef9b617bc234c867f1015 Author: Sunitha Beeram AuthorDate: Wed Mar 6 21:55:17 2019 -0800 Fix formatting issues and figures used --- docs/README.md | 43 --- docs/dev_env.rst | 15 ++- docs/img/PinotBroker.png | Bin 31737 -> 40314 bytes docs/img/PinotController.png | Bin 43621 -> 47991 bytes docs/img/PinotFoundation.png | Bin 35783 -> 44122 bytes docs/img/PinotServer.png | Bin 43540 -> 55062 bytes 6 files changed, 34 insertions(+), 24 deletions(-) diff --git a/docs/README.md b/docs/README.md index 0e41afc..5f23fcc 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,23 +1,28 @@ -.. -.. Licensed to the Apache Software Foundation (ASF) under one -.. or more contributor license agreements. See the NOTICE file -.. distributed with this work for additional information -.. regarding copyright ownership. The ASF licenses this file -.. to you under the Apache License, Version 2.0 (the -.. "License"); you may not use this file except in compliance -.. with the License. You may obtain a copy of the License at -.. -.. http://www.apache.org/licenses/LICENSE-2.0 -.. -.. Unless required by applicable law or agreed to in writing, -.. software distributed under the License is distributed on an -.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -.. KIND, either express or implied. See the License for the -.. specific language governing permissions and limitations -.. under the License. -.. + Updating the docs: 1. Edit or add files as needed. 2. Run "make html" -3. Open _build/html/index.html in your favorite browser and ensure contents and links work correctly. \ No newline at end of file +3. Open _build/html/index.html in your favorite browser and ensure contents and links work correctly. + +NOTE: +You may see some differences locally as the version of sphinx-build on your local host might not be the same as the one used in readthedocs.io for building the pinot website docs. \ No newline at end of file diff --git a/docs/dev_env.rst b/docs/dev_env.rst index d54bb45..dd8aee6 100644 --- a/docs/dev_env.rst +++ b/docs/dev_env.rst @@ -28,17 +28,21 @@ Dev Environment Setup To contribute to Pinot, please follow the instructions below. +Git +--- +Pinot uses git for source code management. If you are new to Git, it will be good to review `basics <https://git-scm.com/book/en/v1/Getting-Started-Git-Basics>`_ of Git and a common tasks like `managing branches <https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell>`_ and `rebasing <https://git-scm.com/book/en/v2/Git-Branching-Rebasing>`_. + Getting the Source Code --- - Create a fork ^ -To limit the number of branches created on the main Pinot repository, we recommend that you create a fork by clicking on the fork button `here <https://github.com/apache/incubator-pinot>`_. - -.. code-block:: none +To limit the number of branches created on the Apache Pinot repository, we recommend that you create a fork by clicking on the fork button `here <https://github.com/apache/incubator-pinot>`_. Clone the repository locally + +.. code-block:: none + $mkdir workspace $cd workspace $git clone https://github.com//incubator-pinot.git @@ -47,9 +51,10 @@ Clone the repository locally Maven - Pinot is a Maven project and familiarity with Maven will help you work with Pinot code. If you are new to Maven, you can -more `here `_ and `here <http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html>`_. +read about Maven `here `_ and `get a quick overview here <http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html>`_. Run the following maven command to setup the project. + .. code-block:: none # compile, download sources diff --git a/docs/img/PinotBroker.png b/docs/img/PinotBroker.png index 675834d..f14badd 100644 Binary files a/docs/img/PinotBroker.png and b/docs/img/PinotBroker.png differ diff --git a/docs/img/PinotController.png b/docs/img/PinotController.png index 451e20b..63a658b 100644 Binary files a/docs/img/PinotController.png and b/docs/img/PinotController.png differ diff --git a/docs/img/PinotFoundation.png b/docs/img/PinotFoundation.png index 889f2fc..91a85a4 100644 Binary files a/docs/img/PinotFoundation.png and b/docs/img/PinotFoundation.png differ diff --git a/docs/img/PinotServer.png b/docs/img/PinotServer.png index f2789bc..38ac17e 100644 Binary files a/docs/img/PinotServer.png and b/docs/img/PinotServer.png differ - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: readthedocs: Fill in dev-guide sections (#3919)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 5feb3a9 readthedocs: Fill in dev-guide sections (#3919)
5feb3a9 is described below
commit 5feb3a9c12d7db1527ad3f2361778ca4aa83703e
Author: Sunitha Beeram
AuthorDate: Wed Mar 6 18:06:07 2019 -0800
readthedocs: Fill in dev-guide sections (#3919)
* Update the display for sidebar navigation to include section names
* Update license headers
* Revert "Update license headers"
This reverts commit 91fb52ae9fc17f0c67b0dccd9f9f0e6dfe1b3ff7.
* Revert licenses for sphinx themes
* Exclude theme files from rat checks
* Remove sphinx theme as there are more licensing implications
* Include updates that were missed in previous commit
* readthedocs: Fill in dev-guide sections
---
docs/{README => README.md} | 0
docs/admin_guide.rst | 1 +
docs/client_api.rst | 6 +-
docs/code_modues.rst | 21 --
docs/code_modules.rst| 141 +++
docs/coding_guidelines.rst | 30 -
docs/contribution_guidelines.rst | 66 ++
docs/dev_env.rst | 58 +---
docs/dev_guide.rst | 2 +-
docs/getting_involved.rst| 21 --
docs/img/PinotBroker.png | Bin 0 -> 31737 bytes
docs/img/PinotController.png | Bin 0 -> 43621 bytes
docs/img/PinotFoundation.png | Bin 0 -> 35783 bytes
docs/img/PinotMinionHadoop.png | Bin 0 -> 38638 bytes
docs/img/PinotServer.png | Bin 0 -> 43540 bytes
docs/img/eclipse_style.png | Bin 0 -> 137578 bytes
docs/img/import_scheme.png | Bin 0 -> 164276 bytes
docs/index.rst | 4 ++
docs/pql_examples.rst| 2 +
19 files changed, 269 insertions(+), 83 deletions(-)
diff --git a/docs/README b/docs/README.md
similarity index 100%
rename from docs/README
rename to docs/README.md
diff --git a/docs/admin_guide.rst b/docs/admin_guide.rst
index be5b6c7..422f56a 100644
--- a/docs/admin_guide.rst
+++ b/docs/admin_guide.rst
@@ -22,5 +22,6 @@
in_production
management_api
+ pinot_hadoop
customizations
tuning_pinot
diff --git a/docs/client_api.rst b/docs/client_api.rst
index cab78ff..e9c1a16 100644
--- a/docs/client_api.rst
+++ b/docs/client_api.rst
@@ -157,8 +157,10 @@ Selection
}
-Java
-
+.. _java-client:
+
+Executing queries via Java Client API
+=
The Pinot client API is similar to JDBC, although there are some differences,
due to how Pinot behaves. For example, a query with multiple aggregation
function will return one result set per aggregation function, as they are
computed in parallel.
diff --git a/docs/code_modues.rst b/docs/code_modues.rst
deleted file mode 100644
index e3a503c..000
--- a/docs/code_modues.rst
+++ /dev/null
@@ -1,21 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-Code Modules and organization
-=
diff --git a/docs/code_modules.rst b/docs/code_modules.rst
new file mode 100644
index 000..adb1fc7
--- /dev/null
+++ b/docs/code_modules.rst
@@ -0,0 +1,141 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the L
[incubator-pinot] branch docs deleted (was 8521b71)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 8521b71 readthedocs: Fill in dev-guide sections The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 02/08: Update license headers
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 05ddfeb5a1635ca5da74e8d20380c8252ee2
Author: Sunitha Beeram
AuthorDate: Tue Mar 5 15:29:26 2019 -0800
Update license headers
---
docs/_themes/sphinx_rtd_theme/__init__.py| 19 +++
docs/_themes/sphinx_rtd_theme/breadcrumbs.html | 20
docs/_themes/sphinx_rtd_theme/footer.html| 20
docs/_themes/sphinx_rtd_theme/layout.html| 20
docs/_themes/sphinx_rtd_theme/search.html| 20
docs/_themes/sphinx_rtd_theme/searchbox.html | 20
.../sphinx_rtd_theme/static/css/badge_only.css | 18 ++
docs/_themes/sphinx_rtd_theme/static/css/theme.css | 18 ++
.../sphinx_rtd_theme/static/js/modernizr.min.js | 18 ++
docs/_themes/sphinx_rtd_theme/static/js/theme.js | 18 ++
docs/_themes/sphinx_rtd_theme/versions.html | 20
11 files changed, 211 insertions(+)
diff --git a/docs/_themes/sphinx_rtd_theme/__init__.py
b/docs/_themes/sphinx_rtd_theme/__init__.py
index b96b5fc..ac2053d 100644
--- a/docs/_themes/sphinx_rtd_theme/__init__.py
+++ b/docs/_themes/sphinx_rtd_theme/__init__.py
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
"""Sphinx ReadTheDocs theme.
From https://github.com/ryan-roemer/sphinx-bootstrap-theme.
diff --git a/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
b/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
index 5578b61..4011c81 100644
--- a/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
+++ b/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
@@ -1,3 +1,23 @@
+
{# Support for Sphinx 1.3+ page_source_suffix, but don't break old builds. #}
{% if page_source_suffix %}
diff --git a/docs/_themes/sphinx_rtd_theme/footer.html
b/docs/_themes/sphinx_rtd_theme/footer.html
index 80815bd..4f56653 100644
--- a/docs/_themes/sphinx_rtd_theme/footer.html
+++ b/docs/_themes/sphinx_rtd_theme/footer.html
@@ -1,3 +1,23 @@
+
{% if (theme_prev_next_buttons_location == 'bottom' or
theme_prev_next_buttons_location == 'both') and (next or prev) %}
diff --git a/docs/_themes/sphinx_rtd_theme/layout.html
b/docs/_themes/sphinx_rtd_theme/layout.html
index cd50ca7..1a2b68a 100644
--- a/docs/_themes/sphinx_rtd_theme/layout.html
+++ b/docs/_themes/sphinx_rtd_theme/layout.html
@@ -1,3 +1,23 @@
+
{# TEMPLATE VAR SETTINGS #}
{%- set url_root = pathto('', 1) %}
{%- if url_root == '#' %}{% set url_root = '' %}{% endif %}
diff --git a/docs/_themes/sphinx_rtd_theme/search.html
b/docs/_themes/sphinx_rtd_theme/search.html
index e3aa9b5..d164694 100644
--- a/docs/_themes/sphinx_rtd_theme/search.html
+++ b/docs/_themes/sphinx_rtd_theme/search.html
@@ -1,3 +1,23 @@
+
{#
basic/search.html
~
diff --git a/docs/_themes/sphinx_rtd_theme/searchbox.html
b/docs/_themes/sphinx_rtd_theme/searchbox.html
index 606f5c8..bbe894d 100644
--- a/docs/_themes/sphinx_rtd_theme/searchbox.html
+++ b/docs/_themes/sphinx_rtd_theme/searchbox.html
@@ -1,3 +1,23 @@
+
{%- if builder != 'singlehtml' %}
diff --git a/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
b/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
index 3c33cef..52d682f 100644
--- a/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
+++ b/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
@@ -1 +1,19 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writin
[incubator-pinot] 08/08: readthedocs: Fill in dev-guide sections
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 8521b719ca521c06a19b26e3773b405bada0196e
Author: Sunitha Beeram
AuthorDate: Wed Mar 6 17:10:36 2019 -0800
readthedocs: Fill in dev-guide sections
---
docs/{README => README.md} | 0
docs/admin_guide.rst | 1 +
docs/client_api.rst | 6 +-
docs/code_modues.rst | 21 --
docs/code_modules.rst| 141 +++
docs/coding_guidelines.rst | 30 -
docs/contribution_guidelines.rst | 66 ++
docs/dev_env.rst | 58 +---
docs/dev_guide.rst | 2 +-
docs/getting_involved.rst| 21 --
docs/img/PinotBroker.png | Bin 0 -> 31737 bytes
docs/img/PinotController.png | Bin 0 -> 43621 bytes
docs/img/PinotFoundation.png | Bin 0 -> 35783 bytes
docs/img/PinotMinionHadoop.png | Bin 0 -> 38638 bytes
docs/img/PinotServer.png | Bin 0 -> 43540 bytes
docs/img/eclipse_style.png | Bin 0 -> 137578 bytes
docs/img/import_scheme.png | Bin 0 -> 164276 bytes
docs/index.rst | 4 ++
docs/pql_examples.rst| 2 +
19 files changed, 269 insertions(+), 83 deletions(-)
diff --git a/docs/README b/docs/README.md
similarity index 100%
rename from docs/README
rename to docs/README.md
diff --git a/docs/admin_guide.rst b/docs/admin_guide.rst
index be5b6c7..422f56a 100644
--- a/docs/admin_guide.rst
+++ b/docs/admin_guide.rst
@@ -22,5 +22,6 @@
in_production
management_api
+ pinot_hadoop
customizations
tuning_pinot
diff --git a/docs/client_api.rst b/docs/client_api.rst
index cab78ff..e9c1a16 100644
--- a/docs/client_api.rst
+++ b/docs/client_api.rst
@@ -157,8 +157,10 @@ Selection
}
-Java
-
+.. _java-client:
+
+Executing queries via Java Client API
+=
The Pinot client API is similar to JDBC, although there are some differences,
due to how Pinot behaves. For example, a query with multiple aggregation
function will return one result set per aggregation function, as they are
computed in parallel.
diff --git a/docs/code_modues.rst b/docs/code_modues.rst
deleted file mode 100644
index e3a503c..000
--- a/docs/code_modues.rst
+++ /dev/null
@@ -1,21 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-Code Modules and organization
-=
diff --git a/docs/code_modules.rst b/docs/code_modules.rst
new file mode 100644
index 000..adb1fc7
--- /dev/null
+++ b/docs/code_modules.rst
@@ -0,0 +1,141 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+.. _code-modules:
+
+Code Modules and Organization
+=
+
+Before proceeding to contributing changes to Pinot, review the contents of
this section.
+
+External Dependencies
+-
+Pinot depends on a number of external projects, the most notable ones are:
+
+* Apache Zookeeper
+* Apache Helix
+* Apache Kafka
+* Apache Thrift
+* Netty
+* Google Guava
+* Yammer
+
+*Helix* is used for ClusterManagement, and Pinot code is tightly integrated
with Helix and Zookeeper interfaces.
+
+*Kafka* is the default realtime stream provider, but can be
[incubator-pinot] 04/08: Revert licenses for sphinx themes
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 7010ec6b7b0778ae7347286db61344bdd245f6d3 Author: Sunitha Beeram AuthorDate: Tue Mar 5 15:56:13 2019 -0800 Revert licenses for sphinx themes --- LICENSE | 1 + licenses/LICENSE-sphinx.txt | 20 pom.xml | 2 ++ 3 files changed, 23 insertions(+) diff --git a/LICENSE b/LICENSE index fcf3095..98d1133 100644 --- a/LICENSE +++ b/LICENSE @@ -218,6 +218,7 @@ pinot-controller/src/main/resources/*/js/lib/handlebars.js pinot-controller/src/main/resources/*/css/lib/codemirror* pinot-controller/src/main/resources/*/css/lib/foundation* pinot-controller/src/main/resources/*/css/lib/normalize.css +docs/_themes/* BSD 3-Clause diff --git a/licenses/LICENSE-sphinx.txt b/licenses/LICENSE-sphinx.txt new file mode 100644 index 000..211dd9c --- /dev/null +++ b/licenses/LICENSE-sphinx.txt @@ -0,0 +1,20 @@ +The MIT License (MIT) + +Copyright (c) 2013-2018 Dave Snider, Read the Docs, Inc. & contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/pom.xml b/pom.xml index d36887d..b087939 100644 --- a/pom.xml +++ b/pom.xml @@ -1071,6 +1071,8 @@ docs/_build/** docs/Makefile + +docs/_themes/** **/*.txt - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 07/08: Include updates that were missed in previous commit
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit bf7c66516d25a15c7bf41e5e2a2e13c15eecddcf Author: Sunitha Beeram AuthorDate: Tue Mar 5 16:42:35 2019 -0800 Include updates that were missed in previous commit --- LICENSE | 1 - licenses/LICENSE-sphinx_rtd_theme.txt | 20 pom.xml | 3 --- 3 files changed, 24 deletions(-) diff --git a/LICENSE b/LICENSE index 98d1133..fcf3095 100644 --- a/LICENSE +++ b/LICENSE @@ -218,7 +218,6 @@ pinot-controller/src/main/resources/*/js/lib/handlebars.js pinot-controller/src/main/resources/*/css/lib/codemirror* pinot-controller/src/main/resources/*/css/lib/foundation* pinot-controller/src/main/resources/*/css/lib/normalize.css -docs/_themes/* BSD 3-Clause diff --git a/licenses/LICENSE-sphinx_rtd_theme.txt b/licenses/LICENSE-sphinx_rtd_theme.txt deleted file mode 100644 index 211dd9c..000 --- a/licenses/LICENSE-sphinx_rtd_theme.txt +++ /dev/null @@ -1,20 +0,0 @@ -The MIT License (MIT) - -Copyright (c) 2013-2018 Dave Snider, Read the Docs, Inc. & contributors - -Permission is hereby granted, free of charge, to any person obtaining a copy of -this software and associated documentation files (the "Software"), to deal in -the Software without restriction, including without limitation the rights to -use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of -the Software, and to permit persons to whom the Software is furnished to do so, -subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS -FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR -COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER -IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN -CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/pom.xml b/pom.xml index f640908..d36887d 100644 --- a/pom.xml +++ b/pom.xml @@ -1071,8 +1071,6 @@ docs/_build/** docs/Makefile - -docs/_themes/** **/*.txt @@ -1149,7 +1147,6 @@ docs/Makefile -docs/_themes/** docs/_build/** - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 03/08: Revert "Update license headers"
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 243c76d422923deef278c9b8cef18e6dc85f7994
Author: Sunitha Beeram
AuthorDate: Tue Mar 5 15:51:27 2019 -0800
Revert "Update license headers"
This reverts commit 91fb52ae9fc17f0c67b0dccd9f9f0e6dfe1b3ff7.
---
docs/_themes/sphinx_rtd_theme/__init__.py| 19 ---
docs/_themes/sphinx_rtd_theme/breadcrumbs.html | 20
docs/_themes/sphinx_rtd_theme/footer.html| 20
docs/_themes/sphinx_rtd_theme/layout.html| 20
docs/_themes/sphinx_rtd_theme/search.html| 20
docs/_themes/sphinx_rtd_theme/searchbox.html | 20
.../sphinx_rtd_theme/static/css/badge_only.css | 18 --
docs/_themes/sphinx_rtd_theme/static/css/theme.css | 18 --
.../sphinx_rtd_theme/static/js/modernizr.min.js | 18 --
docs/_themes/sphinx_rtd_theme/static/js/theme.js | 18 --
docs/_themes/sphinx_rtd_theme/versions.html | 20
11 files changed, 211 deletions(-)
diff --git a/docs/_themes/sphinx_rtd_theme/__init__.py
b/docs/_themes/sphinx_rtd_theme/__init__.py
index ac2053d..b96b5fc 100644
--- a/docs/_themes/sphinx_rtd_theme/__init__.py
+++ b/docs/_themes/sphinx_rtd_theme/__init__.py
@@ -1,22 +1,3 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
"""Sphinx ReadTheDocs theme.
From https://github.com/ryan-roemer/sphinx-bootstrap-theme.
diff --git a/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
b/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
index 4011c81..5578b61 100644
--- a/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
+++ b/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
@@ -1,23 +1,3 @@
-
{# Support for Sphinx 1.3+ page_source_suffix, but don't break old builds. #}
{% if page_source_suffix %}
diff --git a/docs/_themes/sphinx_rtd_theme/footer.html
b/docs/_themes/sphinx_rtd_theme/footer.html
index 4f56653..80815bd 100644
--- a/docs/_themes/sphinx_rtd_theme/footer.html
+++ b/docs/_themes/sphinx_rtd_theme/footer.html
@@ -1,23 +1,3 @@
-
{% if (theme_prev_next_buttons_location == 'bottom' or
theme_prev_next_buttons_location == 'both') and (next or prev) %}
diff --git a/docs/_themes/sphinx_rtd_theme/layout.html
b/docs/_themes/sphinx_rtd_theme/layout.html
index 1a2b68a..cd50ca7 100644
--- a/docs/_themes/sphinx_rtd_theme/layout.html
+++ b/docs/_themes/sphinx_rtd_theme/layout.html
@@ -1,23 +1,3 @@
-
{# TEMPLATE VAR SETTINGS #}
{%- set url_root = pathto('', 1) %}
{%- if url_root == '#' %}{% set url_root = '' %}{% endif %}
diff --git a/docs/_themes/sphinx_rtd_theme/search.html
b/docs/_themes/sphinx_rtd_theme/search.html
index d164694..e3aa9b5 100644
--- a/docs/_themes/sphinx_rtd_theme/search.html
+++ b/docs/_themes/sphinx_rtd_theme/search.html
@@ -1,23 +1,3 @@
-
{#
basic/search.html
~
diff --git a/docs/_themes/sphinx_rtd_theme/searchbox.html
b/docs/_themes/sphinx_rtd_theme/searchbox.html
index bbe894d..606f5c8 100644
--- a/docs/_themes/sphinx_rtd_theme/searchbox.html
+++ b/docs/_themes/sphinx_rtd_theme/searchbox.html
@@ -1,23 +1,3 @@
-
{%- if builder != 'singlehtml' %}
diff --git a/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
b/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
index 52d682f..3c33cef 100644
--- a/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
+++ b/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
@@ -1,19 +1 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.
[incubator-pinot] branch docs created (now 8521b71)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 8521b71 readthedocs: Fill in dev-guide sections This branch includes the following new commits: new 166c77c Update the display for sidebar navigation to include section names new 05ddfeb Update license headers new 243c76d Revert "Update license headers" new 7010ec6 Revert licenses for sphinx themes new 7b10a70 Exclude theme files from rat checks new d011a5a Remove sphinx theme as there are more licensing implications new bf7c665 Include updates that were missed in previous commit new 8521b71 readthedocs: Fill in dev-guide sections The 8 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 05/08: Exclude theme files from rat checks
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 7b10a7004430eacea3a7b5d977518024e072cb7b Author: Sunitha Beeram AuthorDate: Tue Mar 5 16:06:38 2019 -0800 Exclude theme files from rat checks --- pom.xml | 1 + 1 file changed, 1 insertion(+) diff --git a/pom.xml b/pom.xml index b087939..f640908 100644 --- a/pom.xml +++ b/pom.xml @@ -1149,6 +1149,7 @@ docs/Makefile +docs/_themes/** docs/_build/** - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs deleted (was 862aa7e)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 862aa7e readthedocs: Fill in dev-guide sections This change permanently discards the following revisions: discard 862aa7e readthedocs: Fill in dev-guide sections - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs created (now 862aa7e)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 862aa7e readthedocs: Fill in dev-guide sections This branch includes the following new commits: new 862aa7e readthedocs: Fill in dev-guide sections The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: readthedocs: Fill in dev-guide sections
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 862aa7e7bfbc4ad0aa3b56a9b44cb7625b930514
Author: Sunitha Beeram
AuthorDate: Wed Mar 6 17:10:36 2019 -0800
readthedocs: Fill in dev-guide sections
---
docs/{README => README.md} | 0
docs/admin_guide.rst | 1 +
docs/client_api.rst | 6 +-
docs/code_modues.rst | 21 --
docs/code_modules.rst| 141 +++
docs/coding_guidelines.rst | 30 -
docs/contribution_guidelines.rst | 66 ++
docs/dev_env.rst | 58 +---
docs/dev_guide.rst | 2 +-
docs/getting_involved.rst| 21 --
docs/img/PinotBroker.png | Bin 0 -> 31737 bytes
docs/img/PinotController.png | Bin 0 -> 43621 bytes
docs/img/PinotFoundation.png | Bin 0 -> 35783 bytes
docs/img/PinotMinionHadoop.png | Bin 0 -> 38638 bytes
docs/img/PinotServer.png | Bin 0 -> 43540 bytes
docs/img/eclipse_style.png | Bin 0 -> 137578 bytes
docs/img/import_scheme.png | Bin 0 -> 164276 bytes
docs/index.rst | 4 ++
docs/pql_examples.rst| 2 +
19 files changed, 269 insertions(+), 83 deletions(-)
diff --git a/docs/README b/docs/README.md
similarity index 100%
rename from docs/README
rename to docs/README.md
diff --git a/docs/admin_guide.rst b/docs/admin_guide.rst
index be5b6c7..422f56a 100644
--- a/docs/admin_guide.rst
+++ b/docs/admin_guide.rst
@@ -22,5 +22,6 @@
in_production
management_api
+ pinot_hadoop
customizations
tuning_pinot
diff --git a/docs/client_api.rst b/docs/client_api.rst
index cab78ff..e9c1a16 100644
--- a/docs/client_api.rst
+++ b/docs/client_api.rst
@@ -157,8 +157,10 @@ Selection
}
-Java
-
+.. _java-client:
+
+Executing queries via Java Client API
+=
The Pinot client API is similar to JDBC, although there are some differences,
due to how Pinot behaves. For example, a query with multiple aggregation
function will return one result set per aggregation function, as they are
computed in parallel.
diff --git a/docs/code_modues.rst b/docs/code_modues.rst
deleted file mode 100644
index e3a503c..000
--- a/docs/code_modues.rst
+++ /dev/null
@@ -1,21 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-Code Modules and organization
-=
diff --git a/docs/code_modules.rst b/docs/code_modules.rst
new file mode 100644
index 000..adb1fc7
--- /dev/null
+++ b/docs/code_modules.rst
@@ -0,0 +1,141 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+.. _code-modules:
+
+Code Modules and Organization
+=
+
+Before proceeding to contributing changes to Pinot, review the contents of
this section.
+
+External Dependencies
+-
+Pinot depends on a number of external projects, the most notable ones are:
+
+* Apache Zookeeper
+* Apache Helix
+* Apache Kafka
+* Apache Thrift
+* Netty
+* Google Guava
+* Yammer
+
+*Helix* is used for ClusterManagement, and Pinot code is tightly integrated
with Helix and Zookeeper interfaces.
+
+*Kafka* is the default realtime stream provider, but can be
[incubator-pinot] branch docs deleted (was 1976eaf)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 1976eaf Include updates that were missed in previous commit The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Update the display for sidebar navigation to include section names (#3910)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 867edcf Update the display for sidebar navigation to include section
names (#3910)
867edcf is described below
commit 867edcf306cdb9874ba341f0a5c6ff3e286c78f5
Author: Sunitha Beeram
AuthorDate: Tue Mar 5 17:00:03 2019 -0800
Update the display for sidebar navigation to include section names (#3910)
* Update the display for sidebar navigation to include section names
* Update license headers
* Revert "Update license headers"
This reverts commit 91fb52ae9fc17f0c67b0dccd9f9f0e6dfe1b3ff7.
* Revert licenses for sphinx themes
* Exclude theme files from rat checks
* Remove sphinx theme as there are more licensing implications
* Include updates that were missed in previous commit
---
LICENSE | 1 -
docs/conf.py | 14 --
docs/index.rst| 26 --
docs/pql_examples.rst | 2 --
4 files changed, 16 insertions(+), 27 deletions(-)
diff --git a/LICENSE b/LICENSE
index 39683fc..fcf3095 100644
--- a/LICENSE
+++ b/LICENSE
@@ -219,7 +219,6 @@ pinot-controller/src/main/resources/*/css/lib/codemirror*
pinot-controller/src/main/resources/*/css/lib/foundation*
pinot-controller/src/main/resources/*/css/lib/normalize.css
-
BSD 3-Clause
pinot-controller/src/main/resources/*/js/lib/jquery.dataTables.min.js
diff --git a/docs/conf.py b/docs/conf.py
index 7f6d5d2..e34396a 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -31,8 +31,6 @@
# All configuration values have a default; values that are commented out
# serve to show the default.
-import sys
-import os
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
@@ -137,10 +135,14 @@ html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
-#html_theme_options = {}
+html_theme_options = {
+'logo_only': True,
+# Toc options
+'collapse_navigation': False
+ }
# Add any paths that contain custom themes here, relative to this directory.
-#html_theme_path = []
+#html_theme_path = ["_themes",]
# The name for this set of Sphinx documents.
# " v documentation" by default.
@@ -194,13 +196,13 @@ html_static_path = []
#html_split_index = False
# If true, links to the reST sources are added to the pages.
-#html_show_sourcelink = True
+html_show_sourcelink = False
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
-#html_show_copyright = True
+html_show_copyright = False
# If true, an OpenSearch description file will be output, and all pages will
# contain a tag referring to it. The value of this option must be the
diff --git a/docs/index.rst b/docs/index.rst
index a3789f9..728fee5 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -22,42 +22,32 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
-
-Introduction
-
-
+#
+Pinot
+#
.. toctree::
+ :caption: Introduction
:maxdepth: 1
intro
architecture
getting_started
-#
-User Guide
-#
-
.. toctree::
+ :caption: User Guide
:maxdepth: 1
user_guide
-
-#
-Admin Guide
-#
-
.. toctree::
+ :caption: Admin Guide
:maxdepth: 1
admin_guide
-#
-Developer Guide
-#
-
.. toctree::
+ :caption: Developer Guide
:maxdepth: 1
- dev_guide
+ dev_guide
\ No newline at end of file
diff --git a/docs/pql_examples.rst b/docs/pql_examples.rst
index ef69609..8ff4764 100644
--- a/docs/pql_examples.rst
+++ b/docs/pql_examples.rst
@@ -17,8 +17,6 @@
.. under the License.
..
-.. _pql:
-
PQL
===
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs updated: Include updates that were missed in previous commit
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/docs by this push:
new 1976eaf Include updates that were missed in previous commit
1976eaf is described below
commit 1976eaf87ee09ff3aed023e3723b0aa220f12346
Author: Sunitha Beeram
AuthorDate: Tue Mar 5 16:42:35 2019 -0800
Include updates that were missed in previous commit
---
LICENSE | 2 --
docs/conf.py | 5 ++---
licenses/LICENSE-sphinx_rtd_theme.txt | 20
pom.xml | 3 ---
4 files changed, 2 insertions(+), 28 deletions(-)
diff --git a/LICENSE b/LICENSE
index 0d513c3..fcf3095 100644
--- a/LICENSE
+++ b/LICENSE
@@ -218,8 +218,6 @@ pinot-controller/src/main/resources/*/js/lib/handlebars.js
pinot-controller/src/main/resources/*/css/lib/codemirror*
pinot-controller/src/main/resources/*/css/lib/foundation*
pinot-controller/src/main/resources/*/css/lib/normalize.css
-docs/_themes/*
-
BSD 3-Clause
diff --git a/docs/conf.py b/docs/conf.py
index f309406..e34396a 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -138,12 +138,11 @@ html_theme = 'sphinx_rtd_theme'
html_theme_options = {
'logo_only': True,
# Toc options
-'collapse_navigation': False,
-'titles_only': False
+'collapse_navigation': False
}
# Add any paths that contain custom themes here, relative to this directory.
-html_theme_path = ["_themes",]
+#html_theme_path = ["_themes",]
# The name for this set of Sphinx documents.
# " v documentation" by default.
diff --git a/licenses/LICENSE-sphinx_rtd_theme.txt
b/licenses/LICENSE-sphinx_rtd_theme.txt
deleted file mode 100644
index 211dd9c..000
--- a/licenses/LICENSE-sphinx_rtd_theme.txt
+++ /dev/null
@@ -1,20 +0,0 @@
-The MIT License (MIT)
-
-Copyright (c) 2013-2018 Dave Snider, Read the Docs, Inc. & contributors
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of
-this software and associated documentation files (the "Software"), to deal in
-the Software without restriction, including without limitation the rights to
-use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of
-the Software, and to permit persons to whom the Software is furnished to do so,
-subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS
-FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
-COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
-IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
diff --git a/pom.xml b/pom.xml
index f640908..d36887d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1071,8 +1071,6 @@
docs/_build/**
docs/Makefile
-
-docs/_themes/**
**/*.txt
@@ -1149,7 +1147,6 @@
docs/Makefile
-docs/_themes/**
docs/_build/**
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs updated: Exclude theme files from rat checks
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/docs by this push: new e3bd284 Exclude theme files from rat checks e3bd284 is described below commit e3bd284051feb5b856184e0737901c9989754a84 Author: Sunitha Beeram AuthorDate: Tue Mar 5 16:06:38 2019 -0800 Exclude theme files from rat checks --- pom.xml | 1 + 1 file changed, 1 insertion(+) diff --git a/pom.xml b/pom.xml index b087939..f640908 100644 --- a/pom.xml +++ b/pom.xml @@ -1149,6 +1149,7 @@ docs/Makefile +docs/_themes/** docs/_build/** - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs updated (91fb52a -> 7231dc0)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 91fb52a Update license headers new fc4dea0 Revert "Update license headers" new 7231dc0 Revert licenses for sphinx themes The 5794 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: LICENSE | 1 + docs/_themes/sphinx_rtd_theme/__init__.py| 19 --- docs/_themes/sphinx_rtd_theme/breadcrumbs.html | 20 docs/_themes/sphinx_rtd_theme/footer.html| 20 docs/_themes/sphinx_rtd_theme/layout.html| 20 docs/_themes/sphinx_rtd_theme/search.html| 20 docs/_themes/sphinx_rtd_theme/searchbox.html | 20 .../sphinx_rtd_theme/static/css/badge_only.css | 18 -- docs/_themes/sphinx_rtd_theme/static/css/theme.css | 18 -- .../sphinx_rtd_theme/static/js/modernizr.min.js | 18 -- docs/_themes/sphinx_rtd_theme/static/js/theme.js | 18 -- docs/_themes/sphinx_rtd_theme/versions.html | 20 .../LICENSE-sphinx.txt | 19 +-- pom.xml | 2 ++ 14 files changed, 12 insertions(+), 221 deletions(-) copy licenses-binary/LICENSE-normalize.txt => licenses/LICENSE-sphinx.txt (52%) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs updated: Update license headers
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/docs by this push:
new 91fb52a Update license headers
91fb52a is described below
commit 91fb52ae9fc17f0c67b0dccd9f9f0e6dfe1b3ff7
Author: Sunitha Beeram
AuthorDate: Tue Mar 5 15:29:26 2019 -0800
Update license headers
---
docs/_themes/sphinx_rtd_theme/__init__.py| 19 +++
docs/_themes/sphinx_rtd_theme/breadcrumbs.html | 20
docs/_themes/sphinx_rtd_theme/footer.html| 20
docs/_themes/sphinx_rtd_theme/layout.html| 20
docs/_themes/sphinx_rtd_theme/search.html| 20
docs/_themes/sphinx_rtd_theme/searchbox.html | 20
.../sphinx_rtd_theme/static/css/badge_only.css | 18 ++
docs/_themes/sphinx_rtd_theme/static/css/theme.css | 18 ++
.../sphinx_rtd_theme/static/js/modernizr.min.js | 18 ++
docs/_themes/sphinx_rtd_theme/static/js/theme.js | 18 ++
docs/_themes/sphinx_rtd_theme/versions.html | 20
11 files changed, 211 insertions(+)
diff --git a/docs/_themes/sphinx_rtd_theme/__init__.py
b/docs/_themes/sphinx_rtd_theme/__init__.py
index b96b5fc..ac2053d 100644
--- a/docs/_themes/sphinx_rtd_theme/__init__.py
+++ b/docs/_themes/sphinx_rtd_theme/__init__.py
@@ -1,3 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
"""Sphinx ReadTheDocs theme.
From https://github.com/ryan-roemer/sphinx-bootstrap-theme.
diff --git a/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
b/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
index 5578b61..4011c81 100644
--- a/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
+++ b/docs/_themes/sphinx_rtd_theme/breadcrumbs.html
@@ -1,3 +1,23 @@
+
{# Support for Sphinx 1.3+ page_source_suffix, but don't break old builds. #}
{% if page_source_suffix %}
diff --git a/docs/_themes/sphinx_rtd_theme/footer.html
b/docs/_themes/sphinx_rtd_theme/footer.html
index 80815bd..4f56653 100644
--- a/docs/_themes/sphinx_rtd_theme/footer.html
+++ b/docs/_themes/sphinx_rtd_theme/footer.html
@@ -1,3 +1,23 @@
+
{% if (theme_prev_next_buttons_location == 'bottom' or
theme_prev_next_buttons_location == 'both') and (next or prev) %}
diff --git a/docs/_themes/sphinx_rtd_theme/layout.html
b/docs/_themes/sphinx_rtd_theme/layout.html
index cd50ca7..1a2b68a 100644
--- a/docs/_themes/sphinx_rtd_theme/layout.html
+++ b/docs/_themes/sphinx_rtd_theme/layout.html
@@ -1,3 +1,23 @@
+
{# TEMPLATE VAR SETTINGS #}
{%- set url_root = pathto('', 1) %}
{%- if url_root == '#' %}{% set url_root = '' %}{% endif %}
diff --git a/docs/_themes/sphinx_rtd_theme/search.html
b/docs/_themes/sphinx_rtd_theme/search.html
index e3aa9b5..d164694 100644
--- a/docs/_themes/sphinx_rtd_theme/search.html
+++ b/docs/_themes/sphinx_rtd_theme/search.html
@@ -1,3 +1,23 @@
+
{#
basic/search.html
~
diff --git a/docs/_themes/sphinx_rtd_theme/searchbox.html
b/docs/_themes/sphinx_rtd_theme/searchbox.html
index 606f5c8..bbe894d 100644
--- a/docs/_themes/sphinx_rtd_theme/searchbox.html
+++ b/docs/_themes/sphinx_rtd_theme/searchbox.html
@@ -1,3 +1,23 @@
+
{%- if builder != 'singlehtml' %}
diff --git a/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
b/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
index 3c33cef..52d682f 100644
--- a/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
+++ b/docs/_themes/sphinx_rtd_theme/static/css/badge_only.css
@@ -1 +1,19 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may o
[incubator-pinot] branch docs created (now f8158fd)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at f8158fd Update the display for sidebar navigation to include section names This branch includes the following new commits: new f8158fd Update the display for sidebar navigation to include section names The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Remove references that we don't need (#3907)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 0e111e6 Remove references that we don't need (#3907) 0e111e6 is described below commit 0e111e6ab11f3953bf5d94cc7fbfc81248b7f791 Author: Sunitha Beeram AuthorDate: Tue Mar 5 13:01:18 2019 -0800 Remove references that we don't need (#3907) --- docs/admin_guide.rst | 1 - docs/index.rst | 9 - 2 files changed, 10 deletions(-) diff --git a/docs/admin_guide.rst b/docs/admin_guide.rst index ba01ece..be5b6c7 100644 --- a/docs/admin_guide.rst +++ b/docs/admin_guide.rst @@ -22,6 +22,5 @@ in_production management_api - multitenancy customizations tuning_pinot diff --git a/docs/index.rst b/docs/index.rst index 6e809f1..a3789f9 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -61,12 +61,3 @@ Developer Guide :maxdepth: 1 dev_guide - - -Getting Involved - - -.. toctree:: - :maxdepth: 1 - - getting_involved \ No newline at end of file - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs deleted (was d940bbc)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was d940bbc Remove references that we don't need The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Remove references that we don't need
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit d940bbcf1d10c8271814a81cd2fb1d6ac2ff1bc0 Author: Sunitha Beeram AuthorDate: Tue Mar 5 11:15:10 2019 -0800 Remove references that we don't need --- docs/admin_guide.rst | 1 - docs/index.rst | 9 - 2 files changed, 10 deletions(-) diff --git a/docs/admin_guide.rst b/docs/admin_guide.rst index ba01ece..be5b6c7 100644 --- a/docs/admin_guide.rst +++ b/docs/admin_guide.rst @@ -22,6 +22,5 @@ in_production management_api - multitenancy customizations tuning_pinot diff --git a/docs/index.rst b/docs/index.rst index 6e809f1..a3789f9 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -61,12 +61,3 @@ Developer Guide :maxdepth: 1 dev_guide - - -Getting Involved - - -.. toctree:: - :maxdepth: 1 - - getting_involved \ No newline at end of file - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs created (now d940bbc)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at d940bbc Remove references that we don't need This branch includes the following new commits: new d940bbc Remove references that we don't need The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs deleted (was b11dd74)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was b11dd74 Update read-the-docs high level organization. More changes to follow The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Update read-the-docs high level organization. More changes to follow (#3906)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new ec22501 Update read-the-docs high level organization. More changes to
follow (#3906)
ec22501 is described below
commit ec22501cd8ddfa50b09f702cb9852820493ebe17
Author: Sunitha Beeram
AuthorDate: Tue Mar 5 10:21:22 2019 -0800
Update read-the-docs high level organization. More changes to follow (#3906)
---
docs/{reference.rst => README} | 14 --
docs/{reference.rst => admin_guide.rst}| 10 --
docs/{reference.rst => code_modues.rst}| 12 ++--
docs/coding_guidelines.rst | 17 -
docs/{reference.rst => customizations.rst} | 11 ++-
docs/{coding_guidelines.rst => dev_env.rst}| 12 ++--
docs/{reference.rst => dev_guide.rst} | 11 ---
docs/{reference.rst => extensions.rst} | 12 ++--
docs/{reference.rst => getting_involved.rst} | 8
docs/{trying_pinot.rst => getting_started.rst} | 4 ++--
docs/index.rst | 25 +++--
docs/index_techniques.rst | 1 +
docs/{reference.rst => tuning_pinot.rst} | 8 +++-
docs/{reference.rst => user_guide.rst} | 6 +-
14 files changed, 46 insertions(+), 105 deletions(-)
diff --git a/docs/reference.rst b/docs/README
similarity index 82%
copy from docs/reference.rst
copy to docs/README
index bb6891f..0e41afc 100644
--- a/docs/reference.rst
+++ b/docs/README
@@ -17,13 +17,7 @@
.. under the License.
..
-.. _reference:
-
-.. toctree::
- :maxdepth: 1
-
- pql_examples
- index_techniques
- client_api
- management_api
- pinot_hadoop
+Updating the docs:
+1. Edit or add files as needed.
+2. Run "make html"
+3. Open _build/html/index.html in your favorite browser and ensure contents
and links work correctly.
\ No newline at end of file
diff --git a/docs/reference.rst b/docs/admin_guide.rst
similarity index 91%
copy from docs/reference.rst
copy to docs/admin_guide.rst
index bb6891f..ba01ece 100644
--- a/docs/reference.rst
+++ b/docs/admin_guide.rst
@@ -17,13 +17,11 @@
.. under the License.
..
-.. _reference:
-
.. toctree::
:maxdepth: 1
- pql_examples
- index_techniques
- client_api
+ in_production
management_api
- pinot_hadoop
+ multitenancy
+ customizations
+ tuning_pinot
diff --git a/docs/reference.rst b/docs/code_modues.rst
similarity index 86%
copy from docs/reference.rst
copy to docs/code_modues.rst
index bb6891f..e3a503c 100644
--- a/docs/reference.rst
+++ b/docs/code_modues.rst
@@ -17,13 +17,5 @@
.. under the License.
..
-.. _reference:
-
-.. toctree::
- :maxdepth: 1
-
- pql_examples
- index_techniques
- client_api
- management_api
- pinot_hadoop
+Code Modules and organization
+=
diff --git a/docs/coding_guidelines.rst b/docs/coding_guidelines.rst
index 3c415d6..2432276 100644
--- a/docs/coding_guidelines.rst
+++ b/docs/coding_guidelines.rst
@@ -19,23 +19,6 @@
Coding Guidelines
=
-
-To contribute to Pinot, please follow the code-style as described below:
-
-Intellij
-
-
-* `Import <https://www.jetbrains.com/help/idea/settings-code-style.html>`_
code style to Intellij
-* Navigate to ``Preferences`` -> ``Editor`` -> ``Code Style`` -> ``Java``
-* Select ``Import Scheme`` -> ``Intellij IDES code style XML``
-* Choose ``codestyle-intellij.xml``
-
-Eclipse
-^^^
-.. todo::
-
- Add instructions for eclipse here
-
Verifying code-style
diff --git a/docs/reference.rst b/docs/customizations.rst
similarity index 86%
copy from docs/reference.rst
copy to docs/customizations.rst
index bb6891f..3852028 100644
--- a/docs/reference.rst
+++ b/docs/customizations.rst
@@ -17,13 +17,6 @@
.. under the License.
..
-.. _reference:
-.. toctree::
- :maxdepth: 1
-
- pql_examples
- index_techniques
- client_api
- management_api
- pinot_hadoop
+Customization points in Pinot
+=
\ No newline at end of file
diff --git a/docs/coding_guidelines.rst b/docs/dev_env.rst
similarity index 86%
copy from docs/coding_guidelines.rst
copy to docs/dev_env.rst
index 3c415d6..5292703 100644
--- a/docs/coding_guidelines.rst
+++ b/docs/dev_env.rst
@@ -17,8 +17,8 @@
.. under the License.
..
-Coding Guidelines
-=
+Dev Environment Setup
+=
To contribute to Pinot, please follow the code-style as described below:
@@ -36,12 +36,4 @@ Eclipse
Add instructions for eclipse here
-Verifying code-style
-
-
-Run the following command to verify the code-style before posting a PR
-
-.. code
[incubator-pinot] 01/01: Update read-the-docs high level organization. More changes to follow
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch docs
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit b11dd74c08e05ff304f77835bd0b8ae0d85969df
Author: Sunitha Beeram
AuthorDate: Tue Mar 5 09:25:50 2019 -0800
Update read-the-docs high level organization. More changes to follow
---
docs/{reference.rst => README} | 14 --
docs/{reference.rst => admin_guide.rst}| 10 --
docs/{reference.rst => code_modues.rst}| 12 ++--
docs/coding_guidelines.rst | 17 -
docs/{reference.rst => customizations.rst} | 11 ++-
docs/{coding_guidelines.rst => dev_env.rst}| 12 ++--
docs/{reference.rst => dev_guide.rst} | 11 ---
docs/{reference.rst => extensions.rst} | 12 ++--
docs/{reference.rst => getting_involved.rst} | 8
docs/{trying_pinot.rst => getting_started.rst} | 4 ++--
docs/index.rst | 25 +++--
docs/index_techniques.rst | 1 +
docs/{reference.rst => tuning_pinot.rst} | 8 +++-
docs/{reference.rst => user_guide.rst} | 6 +-
14 files changed, 46 insertions(+), 105 deletions(-)
diff --git a/docs/reference.rst b/docs/README
similarity index 82%
copy from docs/reference.rst
copy to docs/README
index bb6891f..0e41afc 100644
--- a/docs/reference.rst
+++ b/docs/README
@@ -17,13 +17,7 @@
.. under the License.
..
-.. _reference:
-
-.. toctree::
- :maxdepth: 1
-
- pql_examples
- index_techniques
- client_api
- management_api
- pinot_hadoop
+Updating the docs:
+1. Edit or add files as needed.
+2. Run "make html"
+3. Open _build/html/index.html in your favorite browser and ensure contents
and links work correctly.
\ No newline at end of file
diff --git a/docs/reference.rst b/docs/admin_guide.rst
similarity index 91%
copy from docs/reference.rst
copy to docs/admin_guide.rst
index bb6891f..ba01ece 100644
--- a/docs/reference.rst
+++ b/docs/admin_guide.rst
@@ -17,13 +17,11 @@
.. under the License.
..
-.. _reference:
-
.. toctree::
:maxdepth: 1
- pql_examples
- index_techniques
- client_api
+ in_production
management_api
- pinot_hadoop
+ multitenancy
+ customizations
+ tuning_pinot
diff --git a/docs/reference.rst b/docs/code_modues.rst
similarity index 86%
copy from docs/reference.rst
copy to docs/code_modues.rst
index bb6891f..e3a503c 100644
--- a/docs/reference.rst
+++ b/docs/code_modues.rst
@@ -17,13 +17,5 @@
.. under the License.
..
-.. _reference:
-
-.. toctree::
- :maxdepth: 1
-
- pql_examples
- index_techniques
- client_api
- management_api
- pinot_hadoop
+Code Modules and organization
+=
diff --git a/docs/coding_guidelines.rst b/docs/coding_guidelines.rst
index 3c415d6..2432276 100644
--- a/docs/coding_guidelines.rst
+++ b/docs/coding_guidelines.rst
@@ -19,23 +19,6 @@
Coding Guidelines
=
-
-To contribute to Pinot, please follow the code-style as described below:
-
-Intellij
-
-
-* `Import <https://www.jetbrains.com/help/idea/settings-code-style.html>`_
code style to Intellij
-* Navigate to ``Preferences`` -> ``Editor`` -> ``Code Style`` -> ``Java``
-* Select ``Import Scheme`` -> ``Intellij IDES code style XML``
-* Choose ``codestyle-intellij.xml``
-
-Eclipse
-^^^
-.. todo::
-
- Add instructions for eclipse here
-
Verifying code-style
diff --git a/docs/reference.rst b/docs/customizations.rst
similarity index 86%
copy from docs/reference.rst
copy to docs/customizations.rst
index bb6891f..3852028 100644
--- a/docs/reference.rst
+++ b/docs/customizations.rst
@@ -17,13 +17,6 @@
.. under the License.
..
-.. _reference:
-.. toctree::
- :maxdepth: 1
-
- pql_examples
- index_techniques
- client_api
- management_api
- pinot_hadoop
+Customization points in Pinot
+=
\ No newline at end of file
diff --git a/docs/coding_guidelines.rst b/docs/dev_env.rst
similarity index 86%
copy from docs/coding_guidelines.rst
copy to docs/dev_env.rst
index 3c415d6..5292703 100644
--- a/docs/coding_guidelines.rst
+++ b/docs/dev_env.rst
@@ -17,8 +17,8 @@
.. under the License.
..
-Coding Guidelines
-=
+Dev Environment Setup
+=
To contribute to Pinot, please follow the code-style as described below:
@@ -36,12 +36,4 @@ Eclipse
Add instructions for eclipse here
-Verifying code-style
-
-
-Run the following command to verify the code-style before posting a PR
-
-.. code-block:: none
-
-mvn checkstyle:check
diff --git a/docs/reference.rst b/docs/dev_guide.rst
similarity index 89%
copy from docs/reference.rst
copy to docs/dev_guide.rst
index bb6891f..6c76e3b 100644
--- a
[incubator-pinot] branch docs created (now b11dd74)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at b11dd74 Update read-the-docs high level organization. More changes to follow This branch includes the following new commits: new b11dd74 Update read-the-docs high level organization. More changes to follow The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch docs deleted (was c6fd5e3)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch docs in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was c6fd5e3 Cleaning conf and fixing last warning The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Replace deprecated ZkClient APIs (#3894)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new caef8f6 Replace deprecated ZkClient APIs (#3894)
caef8f6 is described below
commit caef8f6941250f6723c0a6f847c66122876e1ee8
Author: Hunter Lee
AuthorDate: Thu Feb 28 00:12:54 2019 -0800
Replace deprecated ZkClient APIs (#3894)
The underlying interface for ZkClient in HelixManager has been changed to
HelixZkClient. Using ZkClient directly is no longer recommended by Helix
because it might cause ZkConnection leakage. However, the way Pinot uses
ZkClient here is for testing purposes only, so we need to just replace the
deprecated APIs with updated ones.
---
.../org/apache/pinot/broker/queryquota/TableQueryQuotaManagerTest.java | 2 +-
.../java/org/apache/pinot/broker/routing/TimeBoundaryServiceTest.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/pinot-broker/src/test/java/org/apache/pinot/broker/queryquota/TableQueryQuotaManagerTest.java
b/pinot-broker/src/test/java/org/apache/pinot/broker/queryquota/TableQueryQuotaManagerTest.java
index 25210eb..5ccd243 100644
---
a/pinot-broker/src/test/java/org/apache/pinot/broker/queryquota/TableQueryQuotaManagerTest.java
+++
b/pinot-broker/src/test/java/org/apache/pinot/broker/queryquota/TableQueryQuotaManagerTest.java
@@ -77,7 +77,7 @@ public class TableQueryQuotaManagerTest {
super(clusterName, instanceName, instanceType, zkAddress);
super._zkclient = new ZkClient(StringUtil.join("/",
StringUtils.chomp(ZkStarter.DEFAULT_ZK_STR, "/")),
ZkClient.DEFAULT_SESSION_TIMEOUT,
ZkClient.DEFAULT_CONNECTION_TIMEOUT, new ZNRecordSerializer());
- _zkclient.deleteRecursive("/" + clusterName + "/PROPERTYSTORE");
+ _zkclient.deleteRecursively("/" + clusterName + "/PROPERTYSTORE");
_zkclient.createPersistent("/" + clusterName + "/PROPERTYSTORE", true);
setPropertyStore(clusterName);
}
diff --git
a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/TimeBoundaryServiceTest.java
b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/TimeBoundaryServiceTest.java
index 7f4b200..6058723 100644
---
a/pinot-broker/src/test/java/org/apache/pinot/broker/routing/TimeBoundaryServiceTest.java
+++
b/pinot-broker/src/test/java/org/apache/pinot/broker/routing/TimeBoundaryServiceTest.java
@@ -53,7 +53,7 @@ public class TimeBoundaryServiceTest {
_zkClient = new ZkClient(StringUtil.join("/",
StringUtils.chomp(ZkStarter.DEFAULT_ZK_STR, "/")),
ZkClient.DEFAULT_SESSION_TIMEOUT, ZkClient.DEFAULT_CONNECTION_TIMEOUT,
new ZNRecordSerializer());
String helixClusterName = "TestTimeBoundaryService";
-_zkClient.deleteRecursive("/" + helixClusterName + "/PROPERTYSTORE");
+_zkClient.deleteRecursively("/" + helixClusterName + "/PROPERTYSTORE");
_zkClient.createPersistent("/" + helixClusterName + "/PROPERTYSTORE",
true);
_propertyStore = new ZkHelixPropertyStore<>(new
ZkBaseDataAccessor(_zkClient),
"/" + helixClusterName + "/PROPERTYSTORE", null);
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch postFilterFix deleted (was 4d4a09e)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch postFilterFix in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 4d4a09e Fix reporting/logging of numEntriesScannedPostFilter The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Fix reporting/logging of numEntriesScannedPostFilter (#3870)
This is an automated email from the ASF dual-hosted git repository.
sunithabeeram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new b3f7fee Fix reporting/logging of numEntriesScannedPostFilter (#3870)
b3f7fee is described below
commit b3f7fee22419786b0b9274cc03c7d01bbb1c6b07
Author: Sunitha Beeram
AuthorDate: Fri Feb 22 17:46:35 2019 -0800
Fix reporting/logging of numEntriesScannedPostFilter (#3870)
---
.../main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java
b/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java
index 0003fbe..82d2b5d 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/query/scheduler/QueryScheduler.java
@@ -150,7 +150,7 @@ public abstract class QueryScheduler {
long numEntriesScannedInFilter = Long.parseLong(
dataTableMetadata.getOrDefault(DataTable.NUM_ENTRIES_SCANNED_IN_FILTER_METADATA_KEY,
INVALID_NUM_SCANNED));
long numEntriesScannedPostFilter =
-
Long.parseLong(dataTableMetadata.getOrDefault(DataTable.NUM_SEGMENTS_QUERIED,
INVALID_NUM_SCANNED));
+
Long.parseLong(dataTableMetadata.getOrDefault(DataTable.NUM_ENTRIES_SCANNED_POST_FILTER_METADATA_KEY,
INVALID_NUM_SCANNED));
long numSegmentsProcessed =
Long.parseLong(dataTableMetadata.getOrDefault(DataTable.NUM_SEGMENTS_PROCESSED,
INVALID_SEGMENTS_COUNT));
long numSegmentsMatched =
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch jerseyVersion deleted (was 3b19e29)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to branch jerseyVersion in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 3b19e29 Pick up official release of jersey 2.28 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Pick up official release of jersey 2.28 (#3768)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 006cba0 Pick up official release of jersey 2.28 (#3768) 006cba0 is described below commit 006cba056e4942f13846f550fe18ce3aacb17d1b Author: Sunitha Beeram AuthorDate: Thu Jan 31 11:04:49 2019 -0800 Pick up official release of jersey 2.28 (#3768) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index e4bd807..f7f707c 100644 --- a/pom.xml +++ b/pom.xml @@ -113,7 +113,7 @@ 0.7 2.9.8 1.9.21 -2.28-RC4 +2.28 1.5.10 2.7.0 4.6 - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.017 deleted (was 755a821)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.017 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.017 was deleted! *** tag was 755a821 This change permanently discards the following revisions: discard 879215e updating to 0.017 discard 748d8b3 add pinot-trace - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.015 deleted (was 7554004)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.015 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.015 was deleted! *** tag was 7554004 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.016 deleted (was f503b64)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.016 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.016 was deleted! *** tag was f503b64 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.012 deleted (was aaab747)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.012 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.012 was deleted! *** tag was aaab747 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.014 deleted (was 03eed97)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.014 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.014 was deleted! *** tag was 03eed97 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.013 deleted (was 7d53551)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.013 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.013 was deleted! *** tag was 7d53551 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.01 deleted (was 831c44d)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.01 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.01 was deleted! *** tag was 831c44d The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.007 deleted (was e02b06e)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.007 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.007 was deleted! *** tag was e02b06e The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.008 deleted (was 7ab5dd6)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.008 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.008 was deleted! *** tag was 7ab5dd6 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.006 deleted (was 81d0347)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.006 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.006 was deleted! *** tag was 81d0347 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.009 deleted (was 951bb97)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.009 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.009 was deleted! *** tag was 951bb97 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] annotated tag v0.005 deleted (was 2ddd3a2)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to annotated tag v0.005 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag v0.005 was deleted! *** tag was 2ddd3a2 The revisions that were on this annotated tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] tag mp-VERSION-SNAPSHOT deleted (was e17fc8b)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to tag mp-VERSION-SNAPSHOT in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag mp-VERSION-SNAPSHOT was deleted! *** was e17fc8b Added handlers for llc segment completion protocol This change permanently discards the following revisions: discard e17fc8b Added handlers for llc segment completion protocol discard 77ef97a Adding a class to handle all LLC segment completion requests discard e8526ec Add support for multipart form data for jersey based controller APIs discard 4cda7b0 Commented out an API that does not seem to start right discard cd7b3fd Updated more controller handlers to jersey based framework discard aa3b92a Updated multiple controller API handlers to jersey framework discard 7d07821 Setup jersey based controller admin application - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] tag mp-0.1.98 deleted (was 5b36e76)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to tag mp-0.1.98 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag mp-0.1.98 was deleted! *** was 5b36e76 Added details to IllegalArgumentExceptions thrown The revisions that were on this tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] tag test-tag deleted (was 94beba5)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to tag test-tag in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag test-tag was deleted! *** was 94beba5 dimension dropdownon tabular, refactor js, tabular header tooltip, contributors delta chart area color (#133) The revisions that were on this tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] tag mp-0.1.97 deleted (was 75798e4)
This is an automated email from the ASF dual-hosted git repository. sunithabeeram pushed a change to tag mp-0.1.97 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag mp-0.1.97 was deleted! *** was 75798e4 Fixing compile error on eclipse The revisions that were on this tag are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
