date:20200515

[GitHub] [incubator-pinot] rasin215 opened a new issue #5401: Query Supporting datetime column comparing against currenttime

2020-05-15 Thread GitBox



rasin215 opened a new issue #5401:
URL: https://github.com/apache/incubator-pinot/issues/5401


   Feature Request.
   Most of the time you need a report where we need to compare data against 
current date/time. Currently there is no function which is supporting to 
compare data against current data/time.
   
   e.g select * from table_name where est_date < now()
   
   This query will help to build dashboard for data which is missing eta / 
est_date comparing against current data/time
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] haibow commented on a change in pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox



haibow commented on a change in pull request #5399:
URL: https://github.com/apache/incubator-pinot/pull/5399#discussion_r426114511



##
File path: 
pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java
##
@@ -34,6 +34,8 @@
 import org.apache.pinot.common.utils.CommonConstants;
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.DateTimeFieldSpec;
+import org.apache.pinot.spi.data.DateTimeFormatSpec;

Review comment:
   Please reformat all affected files. This file has unused imports.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] haibow commented on a change in pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox



haibow commented on a change in pull request #5399:
URL: https://github.com/apache/incubator-pinot/pull/5399#discussion_r426114511



##
File path: 
pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java
##
@@ -34,6 +34,8 @@
 import org.apache.pinot.common.utils.CommonConstants;
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.DateTimeFieldSpec;
+import org.apache.pinot.spi.data.DateTimeFormatSpec;

Review comment:
   Please format this file. Has unused imports.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] haibow commented on a change in pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox



haibow commented on a change in pull request #5399:
URL: https://github.com/apache/incubator-pinot/pull/5399#discussion_r426114553



##
File path: 
pinot-core/src/main/java/org/apache/pinot/core/data/function/FunctionEvaluatorFactory.java
##
@@ -59,7 +59,8 @@ public FunctionEvaluator getExpressionEvaluator(FieldSpec 
fieldSpec) {
   }
 } else if (fieldSpec.getFieldType().equals(FieldSpec.FieldType.TIME)) {
 
-  // for backward compatible handling of TIME field conversion
+  // Time conversions should be done using DateTimeFieldSpec and 
treansformFunctions

Review comment:
   nit: s/treansformFunctions/transformFunctions





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] haibow commented on a change in pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox



haibow commented on a change in pull request #5399:
URL: https://github.com/apache/incubator-pinot/pull/5399#discussion_r426114279



##
File path: 
pinot-broker/src/test/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManagerTest.java
##
@@ -75,85 +75,112 @@ public void testTimeBoundaryManager() {
 ExternalView externalView = Mockito.mock(ExternalView.class);
 
 for (TimeUnit timeUnit : TimeUnit.values()) {
-  // Test DAILY push table
+  // Test DAILY push table, with timeFieldSpec
   String rawTableName = "testTable_" + timeUnit + "_DAILY";
-  TableConfig tableConfig = getTableConfig(rawTableName, timeUnit, 
"DAILY");
-  setSchema(rawTableName, timeUnit);
-
-  // Start with no segment
-  TimeBoundaryManager timeBoundaryManager = new 
TimeBoundaryManager(tableConfig, _propertyStore);
-  Set onlineSegments = new HashSet<>();
-  timeBoundaryManager.init(externalView, onlineSegments);
-  assertNull(timeBoundaryManager.getTimeBoundaryInfo());
-
-  // Add the first segment should update the time boundary
-  String segment0 = "segment0";
-  onlineSegments.add(segment0);
-  setSegmentZKMetadata(rawTableName, segment0, 2, timeUnit);
-  timeBoundaryManager.init(externalView, onlineSegments);
-  verifyTimeBoundaryInfo(timeBoundaryManager.getTimeBoundaryInfo(), 
timeUnit.convert(1, TimeUnit.DAYS));
-
-  // Add a new segment with larger end time should update the time boundary
-  String segment1 = "segment1";
-  onlineSegments.add(segment1);
-  setSegmentZKMetadata(rawTableName, segment1, 4, timeUnit);
-  timeBoundaryManager.onExternalViewChange(externalView, onlineSegments);
-  verifyTimeBoundaryInfo(timeBoundaryManager.getTimeBoundaryInfo(), 
timeUnit.convert(3, TimeUnit.DAYS));
-
-  // Add a new segment with smaller end time should not change the time 
boundary
-  String segment2 = "segment2";
-  onlineSegments.add(segment2);
-  setSegmentZKMetadata(rawTableName, segment2, 3, timeUnit);
-  timeBoundaryManager.onExternalViewChange(externalView, onlineSegments);
-  verifyTimeBoundaryInfo(timeBoundaryManager.getTimeBoundaryInfo(), 
timeUnit.convert(3, TimeUnit.DAYS));
-
-  // Remove the segment with largest end time should update the time 
boundary
-  onlineSegments.remove(segment1);
-  timeBoundaryManager.onExternalViewChange(externalView, onlineSegments);
-  verifyTimeBoundaryInfo(timeBoundaryManager.getTimeBoundaryInfo(), 
timeUnit.convert(2, TimeUnit.DAYS));
-
-  // Change segment ZK metadata without refreshing should not update the 
time boundary
-  setSegmentZKMetadata(rawTableName, segment2, 5, timeUnit);
-  timeBoundaryManager.onExternalViewChange(externalView, onlineSegments);
-  verifyTimeBoundaryInfo(timeBoundaryManager.getTimeBoundaryInfo(), 
timeUnit.convert(2, TimeUnit.DAYS));
-
-  // Refresh the changed segment should update the time boundary
-  timeBoundaryManager.refreshSegment(segment2);
-  verifyTimeBoundaryInfo(timeBoundaryManager.getTimeBoundaryInfo(), 
timeUnit.convert(4, TimeUnit.DAYS));
+  TableConfig tableConfig = getTableConfig(rawTableName, "DAILY");
+  setSchemaTimeFieldSpec(rawTableName, timeUnit);
+  testDailyPushTable(rawTableName, tableConfig, timeUnit, externalView);
 
-  // Test HOURLY push table
+  // Test HOURLY push table, with timeFieldSpec
   rawTableName = "testTable_" + timeUnit + "_HOURLY";
-  tableConfig = getTableConfig(rawTableName, timeUnit, "HOURLY");
-  setSchema(rawTableName, timeUnit);
-  timeBoundaryManager = new TimeBoundaryManager(tableConfig, 
_propertyStore);
-  onlineSegments = new HashSet<>();
-  onlineSegments.add(segment0);
-  setSegmentZKMetadata(rawTableName, segment0, 2, timeUnit);
-  timeBoundaryManager.init(externalView, onlineSegments);
-  long expectedTimeValue;
-  if (timeUnit == TimeUnit.DAYS) {
-// Time boundary should be endTime - 1 DAY when time unit is DAYS
-expectedTimeValue = timeUnit.convert(1, TimeUnit.DAYS);
-  } else {
-// Time boundary should be endTime - 1 HOUR when time unit is other 
than DAYS
-expectedTimeValue = timeUnit.convert(47, TimeUnit.HOURS);
-  }
-  verifyTimeBoundaryInfo(timeBoundaryManager.getTimeBoundaryInfo(), 
expectedTimeValue);
+  tableConfig = getTableConfig(rawTableName, "HOURLY");
+  setSchemaTimeFieldSpec(rawTableName, timeUnit);
+  testHourlyPushTable(rawTableName, tableConfig, timeUnit, externalView);
+
+  // Test DAILY push table with dateTimeFieldSpec
+  rawTableName = "testTableDateTime_" + timeUnit + "_DAILY";
+  tableConfig = getTableConfig(rawTableName, "DAILY");
+  setSchemaDateTimeFieldSpec(rawTableName, timeUnit);
+  testDailyPushTable(rawTableName, tableConfig, timeUnit, externalView);
+
+  // Test HOURLY push table
+  rawTableName = "testTableDateTime_" + timeUnit +

[incubator-pinot] branch master updated: Remove timeFieldSpec getter from HybridClusterIntegrationTestRunner (#5397)

2020-05-15 Thread haibow

This is an automated email from the ASF dual-hosted git repository.

haibow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 5dfefe7  Remove timeFieldSpec getter from 
HybridClusterIntegrationTestRunner (#5397)
5dfefe7 is described below

commit 5dfefe71210d1f799cd9d38dd9b7742635716612
Author: Neha Pawar 
AuthorDate: Fri May 15 20:24:19 2020 -0700

Remove timeFieldSpec getter from HybridClusterIntegrationTestRunner (#5397)
---
 .../tests/HybridClusterIntegrationTestCommandLineRunner.java | 12 +++-
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git 
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTestCommandLineRunner.java
 
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTestCommandLineRunner.java
index 9df57ee..b3eec4f 100644
--- 
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTestCommandLineRunner.java
+++ 
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTestCommandLineRunner.java
@@ -39,7 +39,6 @@ import org.apache.pinot.common.segment.ReadMode;
 import org.apache.pinot.controller.ControllerConf;
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.data.Schema;
-import org.apache.pinot.spi.data.TimeFieldSpec;
 import org.apache.pinot.spi.utils.JsonUtils;
 import org.apache.pinot.tools.query.comparison.QueryComparison;
 import org.apache.pinot.util.TestUtils;
@@ -89,14 +88,14 @@ public class HybridClusterIntegrationTestCommandLineRunner {
 
   public static void printUsage() {
 System.err.println(
-"Usage: pinot-hybrid-cluster.sh [--llc] tableName schemaFile dataDir 
invertedIndexColumns sortedColumn");
+"Usage: pinot-hybrid-cluster.sh [--llc] tableName schemaFile 
timeColumnName dataDir invertedIndexColumns sortedColumn");
 System.exit(1);
   }
 
   public static void main(String[] args)
   throws Exception {
 int numArgs = args.length;
-if (!((numArgs == 5) || (numArgs == 6 && args[0].equals("--llc" {
+if (!((numArgs == 6) || (numArgs == 7 && args[0].equals("--llc" {
   printUsage();
 }
 
@@ -111,6 +110,9 @@ public class HybridClusterIntegrationTestCommandLineRunner {
 CustomHybridClusterIntegrationTest._tableName = args[argIdx++];
 File schemaFile = new File(args[argIdx++]);
 CustomHybridClusterIntegrationTest._schema = Schema.fromFile(schemaFile);
+String timeColumnName = args[argIdx++];
+CustomHybridClusterIntegrationTest._timeColumnName =
+
(CustomHybridClusterIntegrationTest._schema.getFieldSpecFor(timeColumnName) != 
null) ? timeColumnName : null;
 File dataDir = new File(args[argIdx++]);
 Preconditions.checkState(dataDir.isDirectory());
 CustomHybridClusterIntegrationTest._dataDir = dataDir;
@@ -179,6 +181,7 @@ public class HybridClusterIntegrationTestCommandLineRunner {
 private static boolean _useLlc = false;
 private static String _tableName;
 private static Schema _schema;
+private static String _timeColumnName;
 private static File _dataDir;
 private static List _invertedIndexColumns;
 private static String _sortedColumn;
@@ -231,8 +234,7 @@ public class HybridClusterIntegrationTestCommandLineRunner {
 @Nullable
 @Override
 protected String getTimeColumnName() {
-  TimeFieldSpec timeFieldSpec = _schema.getTimeFieldSpec();
-  return timeFieldSpec != null ? timeFieldSpec.getName() : null;
+  return _timeColumnName;
 }
 
 @Override


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] haibow merged pull request #5397: Remove timeFieldSpec getter from HybridClusterIntegrationTestRunner

2020-05-15 Thread GitBox



haibow merged pull request #5397:
URL: https://github.com/apache/incubator-pinot/pull/5397


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] haibow commented on pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox



haibow commented on pull request #5399:
URL: https://github.com/apache/incubator-pinot/pull/5399#issuecomment-629580638


   This diff looks like a milestone for TimeFieldSpec deprecation. @npawar 
shall we include this in 0.4.0 release? We could probably hold off on the 
release a bit. cc @kishoreg 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] codecov-io edited a comment on pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox



codecov-io edited a comment on pull request #5399:
URL: https://github.com/apache/incubator-pinot/pull/5399#issuecomment-629578859


   # 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=h1) 
Report
   > Merging 
[#5399](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-pinot/commit/430d48e3fb97ad66905b9fad117fd7de3ad2952e=desc)
 will **increase** coverage by `0.03%`.
   > The diff coverage is `73.68%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-pinot/pull/5399/graphs/tree.svg?width=650=150=pr=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5399  +/-   ##
   ==
   + Coverage   66.81%   66.84%   +0.03% 
   ==
 Files1091 1091  
 Lines   5578355782   -1 
 Branches 8365 8363   -2 
   ==
   + Hits3727337290  +17 
   + Misses  1576215744  -18 
 Partials 2748 2748  
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...he/pinot/controller/util/AutoAddInvertedIndex.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci91dGlsL0F1dG9BZGRJbnZlcnRlZEluZGV4LmphdmE=)
 | `0.00% <0.00%> (ø)` | |
   | 
[...t/core/data/function/FunctionEvaluatorFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL2Z1bmN0aW9uL0Z1bmN0aW9uRXZhbHVhdG9yRmFjdG9yeS5qYXZh)
 | `97.43% <ø> (ø)` | |
   | 
[...va/org/apache/pinot/core/minion/SegmentPurger.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vU2VnbWVudFB1cmdlci5qYXZh)
 | `76.92% <0.00%> (ø)` | |
   | 
[...tion/batch/common/SegmentGenerationTaskRunner.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vcGlub3QtYmF0Y2gtaW5nZXN0aW9uLWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL2luZ2VzdGlvbi9iYXRjaC9jb21tb24vU2VnbWVudEdlbmVyYXRpb25UYXNrUnVubmVyLmphdmE=)
 | `0.00% <0.00%> (ø)` | |
   | 
[...rc/main/java/org/apache/pinot/spi/data/Schema.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9TY2hlbWEuamF2YQ==)
 | `75.08% <ø> (+2.45%)` | :arrow_up: |
   | 
[.../java/org/apache/pinot/spi/data/TimeFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9UaW1lRmllbGRTcGVjLmphdmE=)
 | `88.37% <ø> (ø)` | |
   | 
[...java/org/apache/pinot/spi/utils/TimeConverter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvVGltZUNvbnZlcnRlci5qYXZh)
 | `88.88% <ø> (ø)` | |
   | 
[...oker/routing/timeboundary/TimeBoundaryManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy90aW1lYm91bmRhcnkvVGltZUJvdW5kYXJ5TWFuYWdlci5qYXZh)
 | `89.87% <100.00%> (+0.12%)` | :arrow_up: |
   | 
[...manager/realtime/HLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvSExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh)
 | `82.48% <100.00%> (+0.32%)` | :arrow_up: |
   | 
[...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==)
 | `90.56% <100.00%> (+2.30%)` | :arrow_up: |
   | ... and [36 
more](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=footer).
 Last update 
[430d48e...9cc0d14](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=lastupdated).
 Read

[GitHub] [incubator-pinot] codecov-io commented on pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox



codecov-io commented on pull request #5399:
URL: https://github.com/apache/incubator-pinot/pull/5399#issuecomment-629578859


   # 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=h1) 
Report
   > Merging 
[#5399](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/incubator-pinot/commit/430d48e3fb97ad66905b9fad117fd7de3ad2952e=desc)
 will **increase** coverage by `0.03%`.
   > The diff coverage is `73.68%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-pinot/pull/5399/graphs/tree.svg?width=650=150=pr=4ibza2ugkz)](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#5399  +/-   ##
   ==
   + Coverage   66.81%   66.84%   +0.03% 
   ==
 Files1091 1091  
 Lines   5578355782   -1 
 Branches 8365 8363   -2 
   ==
   + Hits3727337290  +17 
   + Misses  1576215744  -18 
 Partials 2748 2748  
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...he/pinot/controller/util/AutoAddInvertedIndex.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci91dGlsL0F1dG9BZGRJbnZlcnRlZEluZGV4LmphdmE=)
 | `0.00% <0.00%> (ø)` | |
   | 
[...t/core/data/function/FunctionEvaluatorFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL2Z1bmN0aW9uL0Z1bmN0aW9uRXZhbHVhdG9yRmFjdG9yeS5qYXZh)
 | `97.43% <ø> (ø)` | |
   | 
[...va/org/apache/pinot/core/minion/SegmentPurger.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vU2VnbWVudFB1cmdlci5qYXZh)
 | `76.92% <0.00%> (ø)` | |
   | 
[...tion/batch/common/SegmentGenerationTaskRunner.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vcGlub3QtYmF0Y2gtaW5nZXN0aW9uLWNvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcGx1Z2luL2luZ2VzdGlvbi9iYXRjaC9jb21tb24vU2VnbWVudEdlbmVyYXRpb25UYXNrUnVubmVyLmphdmE=)
 | `0.00% <0.00%> (ø)` | |
   | 
[...rc/main/java/org/apache/pinot/spi/data/Schema.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9TY2hlbWEuamF2YQ==)
 | `75.08% <ø> (+2.45%)` | :arrow_up: |
   | 
[.../java/org/apache/pinot/spi/data/TimeFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9UaW1lRmllbGRTcGVjLmphdmE=)
 | `88.37% <ø> (ø)` | |
   | 
[...java/org/apache/pinot/spi/utils/TimeConverter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvVGltZUNvbnZlcnRlci5qYXZh)
 | `88.88% <ø> (ø)` | |
   | 
[...oker/routing/timeboundary/TimeBoundaryManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy90aW1lYm91bmRhcnkvVGltZUJvdW5kYXJ5TWFuYWdlci5qYXZh)
 | `89.87% <100.00%> (+0.12%)` | :arrow_up: |
   | 
[...manager/realtime/HLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvSExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh)
 | `82.48% <100.00%> (+0.32%)` | :arrow_up: |
   | 
[...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==)
 | `90.56% <100.00%> (+2.30%)` | :arrow_up: |
   | ... and [36 
more](https://codecov.io/gh/apache/incubator-pinot/pull/5399/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=footer).
 Last update 
[430d48e...9cc0d14](https://codecov.io/gh/apache/incubator-pinot/pull/5399?src=pr=lastupdated).
 Read the

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5400: Do not release the PinotDataBuffer when closing the index

2020-05-15 Thread GitBox



Jackie-Jiang opened a new pull request #5400:
URL: https://github.com/apache/incubator-pinot/pull/5400


   The PinotDataBuffers are tracked and maintained inside SegmentDirectory
   for ImmutableSegment and PinotDataBufferMemoryManager for MutableSegment.
   They are created when loading the indexes and released when closing the
   segment. If the PinotDataBuffer gets released when closing the index, and
   if the buffer manager decide to reuse the buffer, the following read on
   the buffer will cause JVM to crash. This can be triggered in
   SegmentPreProcessor when the same indexes need to be opened twice in two
   different preprocessors.
   
   This PR standardize the behavior of indexes to not release (close) the
   PinotDataBuffer when closing the index. Also, after the index is closed,
   we should prevent reading anything from the PinotDataBuffer because the
   buffer might already be closed and it could cause JVM to crash.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar opened a new pull request #5399: DATE_TIME should work as the primary time column for Pinot tables

2020-05-15 Thread GitBox



npawar opened a new pull request #5399:
URL: https://github.com/apache/incubator-pinot/pull/5399


   https://github.com/apache/incubator-pinot/issues/2756
   
   This PR ensures that Pinot can use a DateTimeFieldSpec as a primary time 
column for the table.
   
   After this change, we no longer need to use TimeFieldSpec. TimeFieldSpec, 
TimeGranularitySpec have been marked as Deprecated.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] akshayrai opened a new pull request #5398: [TE] Added generic support for configuring Data Quality rules like SLA from Yaml

2020-05-15 Thread GitBox



akshayrai opened a new pull request #5398:
URL: https://github.com/apache/incubator-pinot/pull/5398


   **Overall Updates:**
   * Added ability to configure Data Quality (DQ) alerts from Yaml. See a 
sample below.
   * DQ pipeline leverages & follows the existing detection merging logic 
pattern.
   * Refactored code to modularize & reuse the detection translators and extend 
them to data quality translators.
   * Updated the DATA_QUALITY Task and Pipeline Runner.
   * Added following metrics to track - # DQ Tasks, # DQ Success Tasks & #DQ 
Fail Tasks
   
   Doc Link: To Be Updated
   
   **Tests:**
   * Added a bunch of unit testing around translating DQ yaml to internal json 
representation
   * Updated a bunch of unit testing to validate the Data SLA checker.
   * Tested the Data Quality/sla pipeline by deploying it locally.
   
   **Example:**
   ```
   ...
   rules:
   - quality:
 - type: DATA_SLA
   name: slaRule1
   params:
 sla: 1_DAYS
   ```
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar opened a new pull request #5397: Remove timeFieldSpec getter from HybridClusterIntegrationTestRunner

2020-05-15 Thread GitBox



npawar opened a new pull request #5397:
URL: https://github.com/apache/incubator-pinot/pull/5397


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] kishoreg commented on pull request #4680: Ensure segment converter not fail for schemas without time fieldSpec

2020-05-15 Thread GitBox



kishoreg commented on pull request #4680:
URL: https://github.com/apache/incubator-pinot/pull/4680#issuecomment-629540276


   I think it was to make the behavior similar to batch mode. One has a schema 
that works on offline table and then moves it to real-time and it suddenly 
doesn't work.
   
   Another reason is - there is nothing in Pinot that makes time field 
mandatory. It's just a bunch of columns that one can query just like any other 
database. I  don't see any reason to enforce that
   

   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: The last of the getTimeFieldSpec calls (#5378)

2020-05-15 Thread nehapawar

This is an automated email from the ASF dual-hosted git repository.

nehapawar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 430d48e  The last of the getTimeFieldSpec calls (#5378)
430d48e is described below

commit 430d48e3fb97ad66905b9fad117fd7de3ad2952e
Author: Neha Pawar 
AuthorDate: Fri May 15 15:53:02 2020 -0700

The last of the getTimeFieldSpec calls (#5378)

Removing the last of the getTimeFieldSpec calls.
1. SegmentGeneratorConfig now uses the new getSpecForTimeColumn() call, 
which returns a DateTimeFieldSpec. Time column is expected in table config if 
it is to be considered when creating segment. (this also sets the stage for 
allowing a DateTimeFieldSpec to be a primary time column)
2. Removing TimeFieldSpec special handling from 
RealtimeSegmentConverter.getUpdatedSchema. There is no need to remove the 
incoming time spec. Neither the recordReader nor the dataSource require the 
updated schema. Plus, the record transformer is a pass through.
3. Removing special casing for TimeFieldSpec in 
Schema.isBackwardCompatible() method. The for loop for all specs includes time
---
 .../generator/SegmentGeneratorConfig.java  | 50 ++
 .../minion/rollup/MergeRollupSegmentConverter.java |  1 -
 .../converter/RealtimeSegmentConverter.java| 17 +---
 .../generator/SegmentGeneratorConfigTest.java  | 16 ++-
 .../index/loader/SegmentPreProcessorTest.java  |  2 +-
 .../converter/RealtimeSegmentConverterTest.java| 17 
 .../java/org/apache/pinot/spi/data/Schema.java | 13 +++---
 .../org/apache/pinot/tools/HybridQuickstart.java   | 14 +++---
 .../pinot/tools/streams/AirlineDataStream.java | 12 +++---
 9 files changed, 43 insertions(+), 99 deletions(-)

diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
index 7126ca4..59531fe 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
@@ -31,6 +31,7 @@ import java.util.Set;
 import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 import org.apache.commons.lang.StringUtils;
 import org.apache.pinot.core.io.compression.ChunkCompressorFactory;
 import org.apache.pinot.core.segment.name.FixedSegmentNameGenerator;
@@ -42,6 +43,8 @@ import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.SegmentPartitionConfig;
 import org.apache.pinot.spi.config.table.StarTreeIndexConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.DateTimeFieldSpec;
+import org.apache.pinot.spi.data.DateTimeFormatSpec;
 import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.FieldSpec.FieldType;
 import org.apache.pinot.spi.data.Schema;
@@ -118,10 +121,6 @@ public class SegmentGeneratorConfig {
 setSchema(schema);
 
 // NOTE: SegmentGeneratorConfig#setSchema doesn't set the time column 
anymore. timeColumnName is expected to be read from table config.
-//  If time column name is not set in table config, read time from schema.
-// WARN: Once we move to DateTimeFieldSpec - table config has to be 
provided with valid time - if time needs to be set.
-//  We cannot deduce whether 1) one of the provided DateTimes should be 
used as time column 2) if yes, which one
-//  Even if only 1 DateTime exists, we cannot determine whether it should 
be primary time column (there could be no time column for table (REFRESH), but 
still multiple DateTimeFieldSpec)
 String timeColumnName = null;
 if (tableConfig.getValidationConfig() != null) {
   timeColumnName = tableConfig.getValidationConfig().getTimeColumnName();
@@ -182,42 +181,21 @@ public class SegmentGeneratorConfig {
   }
 
   /**
-   * Set time column details using the given time column. If not found, use 
schema
+   * Set time column details using the given time column
*/
-  public void setTime(String timeColumnName, Schema schema) {
+  public void setTime(@Nullable String timeColumnName, Schema schema) {
 if (timeColumnName != null) {
-  FieldSpec fieldSpec = schema.getFieldSpecFor(timeColumnName);
-  if (fieldSpec != null) {
-setTime(fieldSpec);
-return;
+  DateTimeFieldSpec dateTimeFieldSpec = 
schema.getSpecForTimeColumn(timeColumnName);
+  if (dateTimeFieldSpec != null) {
+setTimeColumnName(dateTimeFieldSpec.getName());
+DateTimeFormatSpec formatSpec = new 
DateTimeFormatSpec(dateTimeFieldSpec.getFormat());
+if

[GitHub] [incubator-pinot] npawar merged pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox



npawar merged pull request #5378:
URL: https://github.com/apache/incubator-pinot/pull/5378


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar commented on pull request #4680: Ensure segment converter not fail for schemas without time fieldSpec

2020-05-15 Thread GitBox



npawar commented on pull request #4680:
URL: https://github.com/apache/incubator-pinot/pull/4680#issuecomment-629538360


   @fx19880617 i know this was a while ago, but do you remember the context of 
why this was done? I thought we don't want to support a realtime table without 
a time column. I think we also have validations in a bunch of places for this. 
Did we decide to make time column non-mandatory?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] mcvsubbu commented on issue #5391: Update segment refresh time in segment metadata on reload

2020-05-15 Thread GitBox



mcvsubbu commented on issue #5391:
URL: 
https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629520902


   > IMO, servers should not change the segment ZK metadata. It should only be 
managed by the controller. We have multiple replicas of the same segment on 
different servers, and segment can be moved to any server at any time. If we 
allow servers to change the ZK metadata, that will be very hard to manage.
   > I think providing API for querying segment metadata is okay. We can return 
the segment metadata for each replica of a segment.
   
   +1
   
   The data of whether servers loaded inverted index (or any other) should be 
with the servers.
   One way of doing this is to raise a metric on the servers when reload fails, 
and set alert on the metric. In theory, the reload can take infinite time to 
arrive and therefore it is never possible to say when it is "done". In 
practice, however, it should be a short period of time for the message itself 
to be transmitted to the servers, so we should be ok



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5391: Update segment refresh time in segment metadata on reload

2020-05-15 Thread GitBox



Jackie-Jiang commented on issue #5391:
URL: 
https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629517968


   IMO, servers should not change the segment ZK metadata. It should only be 
managed by the controller. We have multiple replicas of the same segment on 
different servers, and segment can be moved to any server at any time. If we 
allow servers to change the ZK metadata, that will be very hard to manage.
   I think providing API for querying segment metadata is okay. We can return 
the segment metadata for each replica of a segment.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar commented on issue #5391: Update segment refresh time in segment metadata on reload

2020-05-15 Thread GitBox



npawar commented on issue #5391:
URL: 
https://github.com/apache/incubator-pinot/issues/5391#issuecomment-629515069


   Some context - when users add inverted indexes and reload, we provide no 
indication whether reload was successful or not. This is one of the things that 
can help. Other option is we provide an API for displaying segment metadata 
(from segment dir)- https://github.com/apache/incubator-pinot/issues/5390
   I think both are good to have



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox



Jackie-Jiang commented on a change in pull request #5378:
URL: https://github.com/apache/incubator-pinot/pull/5378#discussion_r426064442



##
File path: 
pinot-core/src/test/java/org/apache/pinot/realtime/converter/RealtimeSegmentConverterTest.java
##
@@ -46,28 +46,11 @@ public void testNoVirtualColumnsInSchema() {
 String segmentName = "segment1";
 
VirtualColumnProviderFactory.addBuiltInVirtualColumnsToSegmentSchema(schema, 
segmentName);
 Assert.assertEquals(schema.getColumnNames().size(), 5);
-
Assert.assertEquals(schema.getTimeFieldSpec().getIncomingGranularitySpec().getTimeType(),
 TimeUnit.MILLISECONDS);
 
 RealtimeSegmentConverter converter =
 new RealtimeSegmentConverter(null, "", schema, "testTable", 
tableConfig, segmentName, "col1");
 
 Schema newSchema = converter.getUpdatedSchema(schema);
 Assert.assertEquals(newSchema.getColumnNames().size(), 2);
-
Assert.assertEquals(newSchema.getTimeFieldSpec().getIncomingGranularitySpec().getTimeType(),
 TimeUnit.DAYS);
-  }
-
-  @Test
-  public void testNoTimeColumnsInSchema() {
-Schema schema = new Schema();

Review comment:
   I think @fx19880617 added that support. Any opinion on this?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: Reduce number of servers in ControllerPeriodicTasksIntegrationTest (#5396)

2020-05-15 Thread jackie

This is an automated email from the ASF dual-hosted git repository.

jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new d7808ac  Reduce number of servers in 
ControllerPeriodicTasksIntegrationTest (#5396)
d7808ac is described below

commit d7808ac5acaf8a4b933cf7564cc334f09fa2404e
Author: Xiaotian (Jackie) Jiang <1751+jackie-ji...@users.noreply.github.com>
AuthorDate: Fri May 15 14:54:06 2020 -0700

Reduce number of servers in ControllerPeriodicTasksIntegrationTest (#5396)

Reduce the number of servers started in 
ControllerPeriodicTasksIntegrationTest to reduce the cost of the test
---
 .../ControllerPeriodicTasksIntegrationTest.java| 40 +-
 1 file changed, 24 insertions(+), 16 deletions(-)

diff --git 
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java
 
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java
index 86bc3a1..1976fa9 100644
--- 
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java
+++ 
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/ControllerPeriodicTasksIntegrationTest.java
@@ -110,8 +110,13 @@ public class ControllerPeriodicTasksIntegrationTest 
extends BaseClusterIntegrati
 }
   }
 
-  private static final int NUM_REPLICAS = 3;
+  private static final int NUM_REPLICAS = 2;
   private static final String TENANT_NAME = "TestTenant";
+  private static final int NUM_BROKERS = 1;
+  private static final int NUM_OFFLINE_SERVERS = 2;
+  private static final int NUM_REALTIME_SERVERS = 2;
+  private static final int NUM_OFFLINE_AVRO_FILES = 8;
+  private static final int NUM_REALTIME_AVRO_FILES = 6;
 
   private String _currentTable = DEFAULT_TABLE_NAME;
 
@@ -151,31 +156,29 @@ public class ControllerPeriodicTasksIntegrationTest 
extends BaseClusterIntegrati
 TestControllerConf controllerConf = new 
TestControllerConf(getDefaultControllerConfiguration());
 controllerConf.setTenantIsolationEnabled(false);
 startController(controllerConf);
-startBroker();
-startServers(6);
+startBrokers(NUM_BROKERS);
+startServers(NUM_OFFLINE_SERVERS + NUM_REALTIME_SERVERS);
 
 // Create tenants
-createBrokerTenant(TENANT_NAME, 1);
-createServerTenant(TENANT_NAME, 3, 3);
+createBrokerTenant(TENANT_NAME, NUM_BROKERS);
+createServerTenant(TENANT_NAME, NUM_OFFLINE_SERVERS, NUM_REALTIME_SERVERS);
 
 // Unpack the Avro files
 int numAvroFiles = unpackAvroData(_tempDir).size();
-int numOfflineAvroFiles = 8;
-int numRealtimeAvroFiles = 6;
 // Avro files has to be ordered as time series data
 List avroFiles = new ArrayList<>(numAvroFiles);
 for (int i = 1; i <= numAvroFiles; i++) {
   avroFiles.add(new File(_tempDir, "On_Time_On_Time_Performance_2014_" + i 
+ ".avro"));
 }
-List offlineAvroFiles = avroFiles.subList(0, numOfflineAvroFiles);
-List realtimeAvroFiles = avroFiles.subList(numAvroFiles - 
numRealtimeAvroFiles, numAvroFiles);
+List offlineAvroFiles = avroFiles.subList(0, NUM_OFFLINE_AVRO_FILES);
+List realtimeAvroFiles = avroFiles.subList(numAvroFiles - 
NUM_REALTIME_AVRO_FILES, numAvroFiles);
 
 // Create and upload the schema and table config
 Schema schema = createSchema();
 addSchema(schema);
 TableConfig offlineTableConfig = createOfflineTableConfig();
 addTableConfig(offlineTableConfig);
-addTableConfig(createRealtimeTableConfig(avroFiles.get(0)));
+addTableConfig(createRealtimeTableConfig(realtimeAvroFiles.get(0)));
 
 // Create and upload segments
 ClusterIntegrationTestUtils
@@ -241,7 +244,7 @@ public class ControllerPeriodicTasksIntegrationTest extends 
BaseClusterIntegrati
 return false;
   }
   if (!checkSegmentStatusCheckerMetrics(controllerMetrics, 
TableNameBuilder.OFFLINE.tableNameWithType(emptyTable),
-  null, 3, 100, 0, 100)) {
+  null, NUM_REPLICAS, 100, 0, 100)) {
 return false;
   }
   if (!checkSegmentStatusCheckerMetrics(controllerMetrics,
@@ -251,17 +254,21 @@ public class ControllerPeriodicTasksIntegrationTest 
extends BaseClusterIntegrati
   }
   String tableNameWithType = 
TableNameBuilder.OFFLINE.tableNameWithType(getTableName());
   IdealState idealState = 
_helixResourceManager.getTableIdealState(tableNameWithType);
-  if (!checkSegmentStatusCheckerMetrics(controllerMetrics, 
tableNameWithType, idealState, 3, 100, 0, 100)) {
+  if (!checkSegmentStatusCheckerMetrics(controllerMetrics, 
tableNameWithType, idealState, NUM_REPLICAS, 100, 0,
+  100)) {
 return false;
   }
   tableNameWithType = 
TableNameBuilder.OFFLINE.tableNameWithType(tableWithOfflineSegment);

[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #5396: Reduce number of servers in ControllerPeriodicTasksIntegrationTest

2020-05-15 Thread GitBox



Jackie-Jiang merged pull request #5396:
URL: https://github.com/apache/incubator-pinot/pull/5396


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar commented on a change in pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox



npawar commented on a change in pull request #5378:
URL: https://github.com/apache/incubator-pinot/pull/5378#discussion_r426058077



##
File path: 
pinot-core/src/test/java/org/apache/pinot/realtime/converter/RealtimeSegmentConverterTest.java
##
@@ -46,28 +46,11 @@ public void testNoVirtualColumnsInSchema() {
 String segmentName = "segment1";
 
VirtualColumnProviderFactory.addBuiltInVirtualColumnsToSegmentSchema(schema, 
segmentName);
 Assert.assertEquals(schema.getColumnNames().size(), 5);
-
Assert.assertEquals(schema.getTimeFieldSpec().getIncomingGranularitySpec().getTimeType(),
 TimeUnit.MILLISECONDS);
 
 RealtimeSegmentConverter converter =
 new RealtimeSegmentConverter(null, "", schema, "testTable", 
tableConfig, segmentName, "col1");
 
 Schema newSchema = converter.getUpdatedSchema(schema);
 Assert.assertEquals(newSchema.getColumnNames().size(), 2);
-
Assert.assertEquals(newSchema.getTimeFieldSpec().getIncomingGranularitySpec().getTimeType(),
 TimeUnit.DAYS);
-  }
-
-  @Test
-  public void testNoTimeColumnsInSchema() {
-Schema schema = new Schema();

Review comment:
   realtime table without time will never happen right?
   Also after changing the getUpdatedSchema method, this test becomes irrelevant





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar commented on a change in pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox



npawar commented on a change in pull request #5378:
URL: https://github.com/apache/incubator-pinot/pull/5378#discussion_r426057601



##
File path: 
pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/RealtimeSegmentConverter.java
##
@@ -156,20 +156,8 @@ public void build(@Nullable SegmentVersion segmentVersion, 
ServerMetrics serverM
   @VisibleForTesting
   public Schema getUpdatedSchema(Schema original) {

Review comment:
   I believe we don't need to change the schema because:
   1) The RecordTransformer is a pass through
   2) The RecordReader is just looping over physical column names so it only 
has outgoing
   
   We shouldn't need to remove incoming, because that implies that we need to 
remove transform functions from all field specs.
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar commented on pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox

npawar commented on pull request #5378:
URL: https://github.com/apache/incubator-pinot/pull/5378#issuecomment-629497281

> Not related to this PR, but inside the Schema can we do the TimeFieldSpec
to DateTimeFieldSpec conversion on the setter side instead of the getter side?
> The problem of conversion on the getter side is that the schema still
stores the deprecated TimeFieldSpec, and `getFieldSpecMap()` and
`getFieldSpecFor()` will still return the TimeFieldSpec instead of
DateTimeFieldSpec. This can easily cause bugs inside the code.

That was exactly the initial plan - to modify the setter to save it as a
DateTimeFieldspec. But now we changed the plan. We are not going to do any
changes to ser/deser. Reason mainly being external integrations (presto,
superset etc) which still need TimeFieldSpec, and also upgrades. I've updated
the design doc:
https://docs.google.com/document/d/1SU1jCjfsIDSA960fD5YWQbD72p8UdGF0c7CroFNt9Ho/edit#heading=h.6gaudq1t6ys8
(Step C and also put the previous approach in Discarded section)

New plan is to make sure Pinot can treat TIME or DATE_TIME either of them as
the primary time column.

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #5396: Reduce number of servers in ControllerPeriodicTasksIntegrationTest

2020-05-15 Thread GitBox



Jackie-Jiang commented on pull request #5396:
URL: https://github.com/apache/incubator-pinot/pull/5396#issuecomment-629473738


   > Is this the cause of the failure?
   
   No, the LinkedIn internal test failure is caused by port already in use. But 
this should be able to fix the issue.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5378: The last of the getTimeFieldSpec calls

2020-05-15 Thread GitBox



Jackie-Jiang commented on a change in pull request #5378:
URL: https://github.com/apache/incubator-pinot/pull/5378#discussion_r426030144



##
File path: 
pinot-core/src/main/java/org/apache/pinot/core/realtime/converter/RealtimeSegmentConverter.java
##
@@ -156,20 +156,8 @@ public void build(@Nullable SegmentVersion segmentVersion, 
ServerMetrics serverM
   @VisibleForTesting
   public Schema getUpdatedSchema(Schema original) {

Review comment:
   We need to remove the time transform here because the time has already 
been transformed.
   Please also update the javadoc

##
File path: 
pinot-core/src/test/java/org/apache/pinot/realtime/converter/RealtimeSegmentConverterTest.java
##
@@ -46,28 +46,11 @@ public void testNoVirtualColumnsInSchema() {
 String segmentName = "segment1";
 
VirtualColumnProviderFactory.addBuiltInVirtualColumnsToSegmentSchema(schema, 
segmentName);
 Assert.assertEquals(schema.getColumnNames().size(), 5);
-
Assert.assertEquals(schema.getTimeFieldSpec().getIncomingGranularitySpec().getTimeType(),
 TimeUnit.MILLISECONDS);
 
 RealtimeSegmentConverter converter =
 new RealtimeSegmentConverter(null, "", schema, "testTable", 
tableConfig, segmentName, "col1");
 
 Schema newSchema = converter.getUpdatedSchema(schema);
 Assert.assertEquals(newSchema.getColumnNames().size(), 2);
-
Assert.assertEquals(newSchema.getTimeFieldSpec().getIncomingGranularitySpec().getTimeType(),
 TimeUnit.DAYS);
-  }
-
-  @Test
-  public void testNoTimeColumnsInSchema() {
-Schema schema = new Schema();

Review comment:
   Do we still want this support? (Realtime table without time column)

##
File path: 
pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
##
@@ -182,42 +180,21 @@ public SegmentGeneratorConfig(TableConfig tableConfig, 
Schema schema) {
   }
 
   /**
-   * Set time column details using the given time column. If not found, use 
schema
+   * Set time column details using the given time column
*/
   public void setTime(String timeColumnName, Schema schema) {

Review comment:
   Annotate `timeColumnName` as nullable or check it on the caller side

##
File path: 
pinot-core/src/test/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfigTest.java
##
@@ -48,14 +48,12 @@ public void testEpochTime() {
 assertNull(segmentGeneratorConfig.getSimpleDateFormat());
 
 // table config not provided

Review comment:
   Remove this line?

##
File path: 
pinot-core/src/test/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfigTest.java
##
@@ -74,14 +72,12 @@ public void testSimpleDateFormat() {
 assertEquals(segmentGeneratorConfig.getSimpleDateFormat(), "MMdd");
 
 // Table config not provided

Review comment:
   Remove this line?

##
File path: pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java
##
@@ -582,10 +582,6 @@ public boolean equals(Object o) {
*/
 
   public boolean isBackwardCompatibleWith(Schema oldSchema) {
-if (!EqualityUtils.isEqual(_timeFieldSpec, oldSchema.getTimeFieldSpec()) 
|| !EqualityUtils
-.isEqual(_dateTimeFieldSpecs, oldSchema.getDateTimeFieldSpecs())) {
-  return false;
-}
 for (Map.Entry entry : 
oldSchema.getFieldSpecMap().entrySet()) {
   if (!getColumnNames().contains(entry.getKey())) {

Review comment:
   Move `getColumnNames()` out of the loop





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5396: Reduce number of servers in ControllerPeriodicTasksIntegrationTest

2020-05-15 Thread GitBox



Jackie-Jiang opened a new pull request #5396:
URL: https://github.com/apache/incubator-pinot/pull/5396


   Reduce the number of servers started in 
ControllerPeriodicTasksIntegrationTest to reduce the cost of the test



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar opened a new pull request #5395: Clean slate on every quickstart run

2020-05-15 Thread GitBox



npawar opened a new pull request #5395:
URL: https://github.com/apache/incubator-pinot/pull/5395


   Using tmp dir generated within quickstart (which has a timestamp) as the 
directory for the zk, controller and server, instead of the default. This will 
ensure we have a clean slate for every quickstart run.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] reallocf commented on issue #5313: Add date time transform functions

2020-05-15 Thread GitBox



reallocf commented on issue #5313:
URL: 
https://github.com/apache/incubator-pinot/issues/5313#issuecomment-629431050


   With https://github.com/apache/incubator-pinot/pull/5326 merging, can we 
close this? Or should we leave it open for some follow-up work?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: Add toDateTime DateTimeFunction (#5326)

2020-05-15 Thread nehapawar

This is an automated email from the ASF dual-hosted git repository.

nehapawar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new e682d49  Add toDateTime DateTimeFunction (#5326)
e682d49 is described below

commit e682d49b2fdb6deadfe69d4efeda2d515cce07b0
Author: Charlie Summers 
AuthorDate: Fri May 15 10:31:04 2020 -0700

Add toDateTime DateTimeFunction (#5326)

Adds toDateTime and fromDateTime inbuilt functions (issue #5313 ).

1) toDateTime takes a long of millis since epoch and a pattern string and 
returns a string corresponding to the DateTime since epoch as the passed 
millis, formatted by the pattern.
2) fromDateTime takes in a DateTime string and a pattern that the DateTime 
string is formatted in and returns a long of millis since epoch corresponding 
to the DateTime string.

Also renamed DefaultFunctionEvaluator to InbuiltFunctionEvaluator and 
FunctionRegistry to InbuiltFunctionRegistry. Adds a FunctionRegistryFactory to 
create InbuiltFunctionRegistrys with a specified set of inbuilt functions. In 
doing so, enables InbuiltFunctionRegistry to register both non-static and 
static functions.

Adds a DateTimePatternHandler that converts DateTime strings to epoch longs 
and epoch longs to DateTime strings backed by a cache of joda 
DateTimeFormatters (each cache is specific to the particular upload).

Adds new tests for toDateTime and fromDateTime and an 
InbuiltFunctionEvaluator test that makes sure the InbuiltFunctionRegistry 
internal state persists between rows for the lifetime of the 
InbuiltFunctionEvaluator.
---
 .../core/data/function/DateTimeFunctions.java  |  16 +++
 .../core/data/function/DateTimePatternHandler.java |  54 +
 .../data/function/FunctionEvaluatorFactory.java|  15 +--
 .../pinot/core/data/function/FunctionRegistry.java | 101 
 .../data/function/FunctionRegistryFactory.java |  81 +
 ...valuator.java => InbuiltFunctionEvaluator.java} |   9 +-
 .../data/function/InbuiltFunctionRegistry.java |  55 +
 .../recordtransformer/ExpressionTransformer.java   |   4 +-
 .../org/apache/pinot/core/util/SchemaUtils.java|   7 +-
 .../function/DateTimeFunctionEvaluatorTest.java|  71 +---
 .../function/DefaultFunctionEvaluatorTest.java | 102 -
 .../function/InbuiltFunctionEvaluatorTest.java | 127 +
 12 files changed, 409 insertions(+), 233 deletions(-)

diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/data/function/DateTimeFunctions.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/data/function/DateTimeFunctions.java
index 24aa4db..2c51197 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/data/function/DateTimeFunctions.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/data/function/DateTimeFunctions.java
@@ -64,6 +64,8 @@ import java.util.concurrent.TimeUnit;
  */
 public class DateTimeFunctions {
 
+  private final DateTimePatternHandler _dateTimePatternHandler = new 
DateTimePatternHandler();
+
   /**
* Convert epoch millis to epoch seconds
*/
@@ -203,4 +205,18 @@ public class DateTimeFunctions {
   static Long fromEpochDaysBucket(Number daysSinceEpoch, Number bucket) {
 return TimeUnit.DAYS.toMillis(daysSinceEpoch.longValue() * 
bucket.intValue());
   }
+
+  /**
+   * Converts epoch millis to DateTime string represented by pattern
+   */
+  String toDateTime(Long millis, String pattern) {
+return _dateTimePatternHandler.parseEpochMillisToDateTimeString(millis, 
pattern);
+  }
+
+  /**
+   * Converts DateTime string represented by pattern to epoch millis
+   */
+  Long fromDateTime(String dateTimeString, String pattern) {
+return 
_dateTimePatternHandler.parseDateTimeStringToEpochMillis(dateTimeString, 
pattern);
+  }
 }
diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/data/function/DateTimePatternHandler.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/data/function/DateTimePatternHandler.java
new file mode 100644
index 000..ae9017d
--- /dev/null
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/data/function/DateTimePatternHandler.java
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS

[GitHub] [incubator-pinot] npawar merged pull request #5326: Add toDateTime DateTimeFunction (#5313)

2020-05-15 Thread GitBox



npawar merged pull request #5326:
URL: https://github.com/apache/incubator-pinot/pull/5326


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: Include dateTimeFieldSpec in schema columns of Pinot Query Console (#5392)

2020-05-15 Thread nehapawar

This is an automated email from the ASF dual-hosted git repository.

nehapawar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 4552477  Include dateTimeFieldSpec in schema columns of Pinot Query 
Console (#5392)
4552477 is described below

commit 45524778e34f4689625fb2e8da4da4fb423662e2
Author: Neha Pawar 
AuthorDate: Fri May 15 10:17:26 2020 -0700

Include dateTimeFieldSpec in schema columns of Pinot Query Console (#5392)
---
 pinot-controller/src/main/resources/static/query/index.html | 6 ++
 pinot-controller/src/main/resources/webapp/index.html   | 6 ++
 2 files changed, 12 insertions(+)

diff --git a/pinot-controller/src/main/resources/static/query/index.html 
b/pinot-controller/src/main/resources/static/query/index.html
index aa142f4..3fc26cd 100644
--- a/pinot-controller/src/main/resources/static/query/index.html
+++ b/pinot-controller/src/main/resources/static/query/index.html
@@ -136,6 +136,12 @@
   {{this.dataType}}
 
   {{/each}}
+  {{#each dateTimeFieldSpecs}}
+  
+{{this.name}}
+{{this.dataType}}
+  
+  {{/each}}
 
   {{#if timeFieldSpec}}
  {{#if outgoingGranularitySpec}}
diff --git a/pinot-controller/src/main/resources/webapp/index.html 
b/pinot-controller/src/main/resources/webapp/index.html
index 5401435..da93b92 100644
--- a/pinot-controller/src/main/resources/webapp/index.html
+++ b/pinot-controller/src/main/resources/webapp/index.html
@@ -131,6 +131,12 @@
   {{this.dataType}}
 
   {{/each}}
+  {{#each dateTimeFieldSpecs}}
+  
+{{this.name}}
+{{this.dataType}}
+  
+  {{/each}}
 
   {{#if timeFieldSpec}}
  {{#if outgoingGranularitySpec}}


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar merged pull request #5392: Include dateTimeFieldSpec in schema columns of Pinot Query Console

2020-05-15 Thread GitBox



npawar merged pull request #5392:
URL: https://github.com/apache/incubator-pinot/pull/5392


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] npawar closed pull request #5369: TimeColumnName and schema to all Integration test tables

2020-05-15 Thread GitBox



npawar closed pull request #5369:
URL: https://github.com/apache/incubator-pinot/pull/5369


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] akshayrai merged pull request #5386: [TE] Add support for configuring additional custom fields in Jira Alert

2020-05-15 Thread GitBox



akshayrai merged pull request #5386:
URL: https://github.com/apache/incubator-pinot/pull/5386


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[incubator-pinot] branch master updated: [TE] Add support for configuring additional custom fields in Jira Alert (#5386)

2020-05-15 Thread akshayrai09

This is an automated email from the ASF dual-hosted git repository.

akshayrai09 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 516a168  [TE] Add support for configuring additional custom fields in 
Jira Alert (#5386)
516a168 is described below

commit 516a16820d7be247c42ff0385654bcecbbaf560b
Author: Akshay Rai 
AuthorDate: Fri May 15 09:48:00 2020 -0700

[TE] Add support for configuring additional custom fields in Jira Alert 
(#5386)
---
 .../dashboard/ThirdEyeDashboardApplication.java|  3 +-
 .../alert/StatefulDetectionAlertFilter.java|  4 +-
 .../alert/scheme/DetectionJiraAlerter.java |  7 ++-
 .../thirdeye/detection/yaml/YamlResource.java  | 51 --
 .../thirdeye/notification/commons/JiraEntity.java  | 15 ++-
 .../notification/commons/ThirdEyeJiraClient.java   | 42 --
 .../formatter/channels/JiraContentFormatter.java   |  2 +
 .../thirdeye/detection/yaml/YamlResourceTest.java  |  2 +-
 .../commons/TestThirdEyeJiraClient.java| 12 +++--
 9 files changed, 110 insertions(+), 28 deletions(-)

diff --git 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/ThirdEyeDashboardApplication.java
 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/ThirdEyeDashboardApplication.java
index 99d8064..1ac0973 100644
--- 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/ThirdEyeDashboardApplication.java
+++ 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/ThirdEyeDashboardApplication.java
@@ -191,7 +191,8 @@ public class ThirdEyeDashboardApplication
 DAO_REGISTRY.getDetectionConfigManager(), 
DAO_REGISTRY.getDetectionAlertConfigManager()));
 env.jersey().register(new DetectionResource());
 env.jersey().register(new 
DetectionAlertResource(DAO_REGISTRY.getDetectionAlertConfigManager()));
-env.jersey().register(new YamlResource(config.getDetectionPreviewConfig(), 
config.getAlertOnboardingPermitPerSecond()));
+env.jersey().register(new YamlResource(config.getAlerterConfiguration(), 
config.getDetectionPreviewConfig(),
+config.getAlertOnboardingPermitPerSecond()));
 env.jersey().register(new SqlDataSourceResource());
 
 TimeSeriesLoader timeSeriesLoader = new DefaultTimeSeriesLoader(
diff --git 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/StatefulDetectionAlertFilter.java
 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/StatefulDetectionAlertFilter.java
index 56c0613..b099677 100644
--- 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/StatefulDetectionAlertFilter.java
+++ 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/StatefulDetectionAlertFilter.java
@@ -48,13 +48,11 @@ public abstract class StatefulDetectionAlertFilter extends 
DetectionAlertFilter
   public static final String PROP_BCC = "bcc";
   public static final String PROP_RECIPIENTS = "recipients";
 
-  private static final String PROP_SEND_ONCE = "sendOnce";
-
   // Time beyond which we do not want to notify anomalies
   private static final long ANOMALY_NOTIFICATION_LOOKBACK_TIME = 
TimeUnit.DAYS.toMillis(14);
 
   public StatefulDetectionAlertFilter(DataProvider provider, 
DetectionAlertConfigDTO config, long endTime) {
-super(provider, config, 
endTime);//prepareStatement.setObject(parameterIndex++, pair.getValue(), 
info.sqlType);
+super(provider, config, endTime);
   }
 
   protected final Set filter(Map 
vectorClocks) {
diff --git 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/scheme/DetectionJiraAlerter.java
 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/scheme/DetectionJiraAlerter.java
index 70d6fc2..4d2c7bc 100644
--- 
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/scheme/DetectionJiraAlerter.java
+++ 
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/scheme/DetectionJiraAlerter.java
@@ -63,6 +63,9 @@ import static 
org.apache.pinot.thirdeye.notification.commons.JiraConfiguration.*
  * labels:  # optional, default - thirdeye label is always 
appended
  *   - test-label-1
  *   - test-label-2
+ * custom:
+ *   test1: value1
+ *   test2: value2
  */
 @AlertScheme(type = "JIRA")
 public class DetectionJiraAlerter extends DetectionAlertScheme {
@@ -92,7 +95,7 @@ public class DetectionJiraAlerter extends 
DetectionAlertScheme {
   }
 
   private void updateJiraAlert(Issue issue, JiraEntity jiraEntity) {
-// Append labels
+// Append labels - do not remove existing labels
 jiraEntity.getLabels().addAll(issue.getLabels());

[GitHub] [incubator-pinot] jamesyfshao opened a new pull request #5394: add callback interface for upsert component

2020-05-15 Thread GitBox



jamesyfshao opened a new pull request #5394:
URL: https://github.com/apache/incubator-pinot/pull/5394


   added list of critical interface that we need to inject to pinot core/server 
to ensure we can add proper logic for upsert-enabled pinotserver and tables and 
making sure existing pinot tables perform the same as before. Pinot upsert 
related logics will be implemented in separate package and injected to runtime 
environment via config. This diff also provided a no-op implementation to 
ensure that existing pinot ingestion workflow stay the same
   
   design doc for pinot upsert feature: [google 
doc](https://docs.google.com/document/d/1SFFir7ByxCff-aVYxQeTHpNhPXeP5q7P4g_6O2iNGgU/edit?usp=sharing)
 (our corp privacy policy disallow blanket sharing so please send a 
view/comment request to me and I usually accept it as soon as I see it)
   
   this change focus on the changes in pinot core/server



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[GitHub] [incubator-pinot] xiaohui-sun opened a new issue #5393: [TE] Let's put this name in configuration

2020-05-15 Thread GitBox



xiaohui-sun opened a new issue #5393:
URL: https://github.com/apache/incubator-pinot/issues/5393


   
https://github.com/apache/incubator-pinot/blob/a1401de18c10fc07b557a6e5ce0efacd8d486869/thirdeye/thirdeye-frontend/app/pods/home/index/route.js#L81
   
   @harleyjj 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

41 matches

Mail list logo