[GitHub] [incubator-pinot] fx19880617 edited a comment on pull request #5456: Single Quote String Literal for function invoker results
fx19880617 edited a comment on pull request #5456: URL: https://github.com/apache/incubator-pinot/pull/5456#issuecomment-635100955 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on pull request #5456: Single Quote String Literal for function invoker results
fx19880617 commented on pull request #5456: URL: https://github.com/apache/incubator-pinot/pull/5456#issuecomment-635100955 > I think the real issue is BrokerRequest is incapable of storing expressions correctly, it converts them to String. This seems like a temp workaround (approving it on that pretext). The ideal fix is to move to PinotQuery. Correct, this is also mention by @kishoreg , I will soon started move query processing from brokerRequest to PinotQuery. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg merged pull request #5456: Single Quote String Literal for function invoker results
kishoreg merged pull request #5456: URL: https://github.com/apache/incubator-pinot/pull/5456 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Single Quote String Literal for function invoker results (#5456)
This is an automated email from the ASF dual-hosted git repository.
kishoreg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 7f10c5c Single Quote String Literal for function invoker results
(#5456)
7f10c5c is described below
commit 7f10c5c9a571ba1ca8d8ebee5d2b74fc3b05cd7b
Author: Xiang Fu
AuthorDate: Wed May 27 21:55:14 2020 -0700
Single Quote String Literal for function invoker results (#5456)
---
.../org/apache/pinot/sql/parsers/CalciteSqlParser.java | 3 +++
.../apache/pinot/sql/parsers/CalciteSqlCompilerTest.java | 6 +++---
.../integration/tests/OfflineClusterIntegrationTest.java | 14 ++
3 files changed, 20 insertions(+), 3 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
index 45b3274..2aec8ec 100644
---
a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
+++
b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
@@ -676,6 +676,9 @@ public class CalciteSqlParser {
try {
FunctionInvoker invoker = new FunctionInvoker(functionInfo);
Object result = invoker.process(arguments);
+if (result instanceof String) {
+ result = String.format("'%s'", result);
+}
return RequestUtils.getLiteralExpression(result);
} catch (Exception e) {
throw new SqlCompilationException(new
IllegalArgumentException("Unsupported function - " + funcName, e));
diff --git
a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
index ac0a5cf..aec7917 100644
---
a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
+++
b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
@@ -1534,7 +1534,7 @@ public class CalciteSqlCompilerTest {
Function greaterThan = pinotQuery.getFilterExpression().getFunctionCall();
String today =
greaterThan.getOperands().get(1).getLiteral().getStringValue();
String expectedTodayStr =
-
Instant.now().atZone(ZoneId.of("UTC")).format(DateTimeFormatter.ofPattern("-MM-dd
z"));
+"'" +
Instant.now().atZone(ZoneId.of("UTC")).format(DateTimeFormatter.ofPattern("-MM-dd
z")) + "'";
Assert.assertEquals(today, expectedTodayStr);
}
@@ -1557,7 +1557,7 @@ public class CalciteSqlCompilerTest {
Assert.assertTrue(expression.getLiteral() != null);
String today = expression.getLiteral().getStringValue();
String expectedTodayStr =
-
Instant.now().atZone(ZoneId.of("UTC")).format(DateTimeFormatter.ofPattern("-MM-dd
z"));
+"'" +
Instant.now().atZone(ZoneId.of("UTC")).format(DateTimeFormatter.ofPattern("-MM-dd
z")) + "'";
Assert.assertEquals(today, expectedTodayStr);
expression =
CalciteSqlParser.compileToExpression("toDateTime(playerName)");
Assert.assertTrue(expression.getFunctionCall() != null);
@@ -1575,7 +1575,7 @@ public class CalciteSqlCompilerTest {
Assert.assertTrue(expression.getFunctionCall() != null);
expression =
CalciteSqlParser.invokeCompileTimeFunctionExpression(expression);
Assert.assertTrue(expression.getLiteral() != null);
-Assert.assertEquals(expression.getLiteral().getFieldValue(), "emaNreyalp");
+Assert.assertEquals(expression.getLiteral().getFieldValue(),
"'emaNreyalp'");
expression = CalciteSqlParser.compileToExpression("count(*)");
Assert.assertTrue(expression.getFunctionCall() != null);
expression =
CalciteSqlParser.invokeCompileTimeFunctionExpression(expression);
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
index f433a3e..223602c 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
@@ -24,6 +24,9 @@ import com.fasterxml.jackson.databind.node.ObjectNode;
import com.google.common.collect.ImmutableList;
import java.io.File;
import java.io.IOException;
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
@@ -309,6 +312,17 @@ public class OfflineClusterIntegrationTest extends
BaseClusterIntegrationTestSet
}
@Test
+ public void testTimeFunc()
+ throws Exception {
+String sqlQuery = "SELECT toDateTime(now(), '-MM-dd z') FROM mytable";
+
[GitHub] [incubator-pinot] mayankshriv edited a comment on pull request #5456: Single Quote String Literal for function invoker results
mayankshriv edited a comment on pull request #5456: URL: https://github.com/apache/incubator-pinot/pull/5456#issuecomment-635099149 I think the real issue is BrokerRequest is incapable of storing expressions correctly, it converts them to String. This seems like a temp workaround (approving it on that pretext). The ideal fix is to move to PinotQuery. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv commented on pull request #5456: Single Quote String Literal for function invoker results
mayankshriv commented on pull request #5456: URL: https://github.com/apache/incubator-pinot/pull/5456#issuecomment-635099149 I think the real issue is BrokerRequest is incapable of storing expressions correctly, it converts them to String. This seems like a temp workaround (approving it on that pretext). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] tag release-0.4.0 created (now 3f81191)
This is an automated email from the ASF dual-hosted git repository. haibow pushed a change to tag release-0.4.0 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 3f81191 (commit) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai merged pull request #5458: [TE] clean up some code - moved task creation methods under util
akshayrai merged pull request #5458: URL: https://github.com/apache/incubator-pinot/pull/5458 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (4988d72 -> b6cb44c)
This is an automated email from the ASF dual-hosted git repository. akshayrai09 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 4988d72 [TE] add back frontend as submodule (#5457) add b6cb44c [TE] clean up somce code and disable sla alerts till we have the ramp feature (#5458) No new revisions were added by this update. Summary of changes: .../trigger/DataAvailabilityTaskScheduler.java | 38 +- .../apache/pinot/thirdeye/detection/TaskUtils.java | 31 +- .../dataquality/DataQualityPipelineJob.java| 14 +--- 3 files changed, 40 insertions(+), 43 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull request #5435: [TE] clean up decprecated/unused code
vincentchenjl commented on a change in pull request #5435:
URL: https://github.com/apache/incubator-pinot/pull/5435#discussion_r431529266
##
File path:
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/AnomalyResource.java
##
@@ -101,21 +99,18 @@
private MergedAnomalyResultManager anomalyMergedResultDAO;
private AlertConfigManager emailConfigurationDAO;
private MergedAnomalyResultManager mergedAnomalyResultDAO;
- private AutotuneConfigManager autotuneConfigDAO;
private DatasetConfigManager datasetConfigDAO;
private AnomalyFunctionFactory anomalyFunctionFactory;
private AlertFilterFactory alertFilterFactory;
private LoadingCache collectionMaxDataTimeCache;
private static final DAORegistry DAO_REGISTRY = DAORegistry.getInstance();
- public AnomalyResource(AnomalyFunctionFactory anomalyFunctionFactory,
AlertFilterFactory alertFilterFactory,
- AlertFilterAutotuneFactory alertFilterAutotuneFactory) {
+ public AnomalyResource(AnomalyFunctionFactory anomalyFunctionFactory,
AlertFilterFactory alertFilterFactory) {
this.anomalyFunctionDAO = DAO_REGISTRY.getAnomalyFunctionDAO();
Review comment:
`anomalyFunctionDAO` is used in multiple places in this class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull request #5435: [TE] clean up decprecated/unused code
vincentchenjl commented on a change in pull request #5435:
URL: https://github.com/apache/incubator-pinot/pull/5435#discussion_r431528717
##
File path:
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/datalayer/DaoTestUtils.java
##
@@ -275,22 +273,6 @@ public static DetectionStatusDTO
getTestDetectionStatus(String dataset, long dat
return detectionStatusDTO;
}
- public static AutotuneConfigDTO getTestAutotuneConfig(long functionId, long
start, long end) {
-AutotuneConfigDTO autotuneConfigDTO = new AutotuneConfigDTO();
-autotuneConfigDTO.setFunctionId(functionId);
-autotuneConfigDTO.setStartTime(start);
-autotuneConfigDTO.setEndTime(end);
-
autotuneConfigDTO.setPerformanceEvaluationMethod(PerformanceEvaluationMethod.ANOMALY_PERCENTAGE);
-autotuneConfigDTO.setLastUpdateTimestamp(DateTime.now().getMillis());
-Map config = new HashMap<>();
-config.put("ConfigKey", "ConfigValue");
-autotuneConfigDTO.setConfiguration(config);
-Map performance = new HashMap<>();
-performance.put(autotuneConfigDTO.getPerformanceEvaluationMethod().name(),
0.5);
-autotuneConfigDTO.setPerformance(performance);
-return autotuneConfigDTO;
- }
-
public static ClassificationConfigDTO getTestClassificationConfig(String
name, List mainFunctionIdList,
Review comment:
This can be removed in next PR with all the unused DAO objects.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull request #5435: [TE] clean up decprecated/unused code
vincentchenjl commented on a change in pull request #5435:
URL: https://github.com/apache/incubator-pinot/pull/5435#discussion_r431528040
##
File path:
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/AnomalyResource.java
##
@@ -101,21 +99,18 @@
private MergedAnomalyResultManager anomalyMergedResultDAO;
private AlertConfigManager emailConfigurationDAO;
private MergedAnomalyResultManager mergedAnomalyResultDAO;
- private AutotuneConfigManager autotuneConfigDAO;
private DatasetConfigManager datasetConfigDAO;
private AnomalyFunctionFactory anomalyFunctionFactory;
private AlertFilterFactory alertFilterFactory;
private LoadingCache collectionMaxDataTimeCache;
private static final DAORegistry DAO_REGISTRY = DAORegistry.getInstance();
- public AnomalyResource(AnomalyFunctionFactory anomalyFunctionFactory,
AlertFilterFactory alertFilterFactory,
- AlertFilterAutotuneFactory alertFilterAutotuneFactory) {
+ public AnomalyResource(AnomalyFunctionFactory anomalyFunctionFactory,
AlertFilterFactory alertFilterFactory) {
this.anomalyFunctionDAO = DAO_REGISTRY.getAnomalyFunctionDAO();
this.anomalyMergedResultDAO = DAO_REGISTRY.getMergedAnomalyResultDAO();
Review comment:
Both these two are used in the class. If we r sure `AnomalyResource` is
not used, we can remove the whole class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull request #5435: [TE] clean up decprecated/unused code
vincentchenjl commented on a change in pull request #5435: URL: https://github.com/apache/incubator-pinot/pull/5435#discussion_r431527731 ## File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/ThirdEyeDashboardApplication.java ## @@ -160,14 +155,12 @@ public void run(ThirdEyeDashboardConfiguration config, Environment env) AnomalyFunctionFactory anomalyFunctionFactory = new AnomalyFunctionFactory(config.getFunctionConfigPath()); AlertFilterFactory alertFilterFactory = new AlertFilterFactory(config.getAlertFilterConfigPath()); Review comment: Same as above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull request #5435: [TE] clean up decprecated/unused code
vincentchenjl commented on a change in pull request #5435: URL: https://github.com/apache/incubator-pinot/pull/5435#discussion_r431526938 ## File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/ThirdEyeDashboardApplication.java ## @@ -160,14 +155,12 @@ public void run(ThirdEyeDashboardConfiguration config, Environment env) AnomalyFunctionFactory anomalyFunctionFactory = new AnomalyFunctionFactory(config.getFunctionConfigPath()); Review comment: `anomalyFunctionFactory` is used to initialize `AnomalyResource`, `DataResource`, and `AnomaliesResource`. Could we confirm all these three resources are not used? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull request #5435: [TE] clean up decprecated/unused code
vincentchenjl commented on a change in pull request #5435: URL: https://github.com/apache/incubator-pinot/pull/5435#discussion_r431522638 ## File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/task/TaskConstants.java ## @@ -26,15 +26,8 @@ DETECTION, DETECTION_ALERT, YAML_DETECTION_ONBOARD, -ANOMALY_DETECTION, -MERGE, // TODO: deprecate ALERT task type -ALERT, -ALERT2, -MONITOR, -DATA_COMPLETENESS, -CLASSIFICATION, -REPLAY +MONITOR Review comment: Added one line description each type. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5459: [Cleanup] Merge RealtimeSegmentOnlineOfflineStateModel and SegmentOnlineOfflineStateModel in CommonConstants
Jackie-Jiang opened a new pull request #5459: URL: https://github.com/apache/incubator-pinot/pull/5459 We only have one SegmentOnlineOfflineStateModel, so there is no value keeping both of them ## Description Add a description of your PR here. A good description should include pointers to an issue or design document, etc. ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) * [ ] Yes (Please label as **backward-incompat**, and complete the section below on Release Notes) Does this PR fix a zero-downtime upgrade introduced earlier? * [ ] Yes (Please label this as **backward-incompat**, and complete the section below on Release Notes) Does this PR otherwise need attention when creating release notes? Things to consider: - New configuration options - Deprecation of configurations - Signature changes to public methods/interfaces - New plugins added or old plugins removed * [ ] Yes (Please label this PR as **release-notes** and complete the section on Release Notes) ## Release Notes If you have tagged this as either backward-incompat or release-notes, you MUST add text here that you would like to see appear in release notes of the next release. If you have a series of commits adding or enabling a feature, then add this section only in final commit that marks the feature completed. Refer to earlier release notes to see examples of text ## Documentation If you have introduced a new feature or configuration, please add it to the documentation as well. See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai opened a new pull request #5458: [TE] clean up some code and disable sla alerts till we have the ramp feature
akshayrai opened a new pull request #5458: URL: https://github.com/apache/incubator-pinot/pull/5458 * Move all the task creation methods under TaskUtils * Disable SLA alerts and wait till the experiment/ramp feature This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai merged pull request #5457: [TE] add back frontend as submodule
akshayrai merged pull request #5457: URL: https://github.com/apache/incubator-pinot/pull/5457 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (d8ea89c -> 4988d72)
This is an automated email from the ASF dual-hosted git repository. akshayrai09 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from d8ea89c Create PULL_REQUEST_TEMPLATE.md (#5345) add 4988d72 [TE] add back frontend as submodule (#5457) No new revisions were added by this update. Summary of changes: .travis/.travis_te_nightly_build.sh | 4 ++-- thirdeye/pom.xml| 1 + thirdeye/thirdeye-frontend/package.json | 2 +- thirdeye/thirdeye-frontend/pom.xml | 22 -- 4 files changed, 24 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl opened a new pull request #5457: [TE] add back frontend as submodule
vincentchenjl opened a new pull request #5457: URL: https://github.com/apache/incubator-pinot/pull/5457 This PR is to achieve the following items. - Fix open source build by adding back frontend as submodule. - Ensure that frontend is built based on packages from open source NPM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu merged pull request #5345: Create PULL_REQUEST_TEMPLATE.md
mcvsubbu merged pull request #5345: URL: https://github.com/apache/incubator-pinot/pull/5345 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (cee2b23 -> d8ea89c)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from cee2b23 [TE] add mock events loader to pinot-thirdeye docker defaults (#5454) add d8ea89c Create PULL_REQUEST_TEMPLATE.md (#5345) No new revisions were added by this update. Summary of changes: .github/PULL_REQUEST_TEMPLATE.md | 28 .gitignore | 4 +++- 2 files changed, 31 insertions(+), 1 deletion(-) create mode 100644 .github/PULL_REQUEST_TEMPLATE.md - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on pull request #5456: Single Quote String Literal for function invoker results
fx19880617 commented on pull request #5456:
URL: https://github.com/apache/incubator-pinot/pull/5456#issuecomment-634954131
> looks like this fix is a workaround to hide the actual problem in
transform function?
>
> What would be the exception/error in transformfunction without this fix
There is no error, in brokerRequest, it's converted to a string. E.g. if the
literal is `2020-05-27`, without single quote, transform function will parse it
to `2020` only.
See here :
TransformExpressionTree.compileToExpressionTree("2020-05-27 UTC") = '2020'
TransformExpressionTree.compileToExpressionTree("'2020-05-27 UTC'") =
'2020-05-27 UTC'
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on pull request #5456: Single Quote String Literal for function invoker results
kishoreg commented on pull request #5456: URL: https://github.com/apache/incubator-pinot/pull/5456#issuecomment-634948647 looks like this fix is a workaround to hide the actual problem in transform function? What would be the exception/error in transformfunction without this fix This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5456: Single Quote String Literal for function invoker results
fx19880617 opened a new pull request #5456: URL: https://github.com/apache/incubator-pinot/pull/5456 This fix the issue that of function invoker string output breaks query syntax. E.g. query: `SELECT toDateTime(now(), '-MM-dd z') FROM mytable` becomes: `SELECT 2020-05-27 UTC FROM mytable`. And not recognized in Transformation function. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch single_quote_string_literal_for_function_invoke_result created (now 948062a)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch single_quote_string_literal_for_function_invoke_result in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 948062a Single Quote String Literal for function invoker results This branch includes the following new commits: new 948062a Single Quote String Literal for function invoker results The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Single Quote String Literal for function invoker results
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch
single_quote_string_literal_for_function_invoke_result
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 948062aa5f82a2fb508b5f3990e64bd10ccfa553
Author: Xiang Fu
AuthorDate: Wed May 27 14:14:57 2020 -0700
Single Quote String Literal for function invoker results
---
.../org/apache/pinot/sql/parsers/CalciteSqlParser.java | 3 +++
.../apache/pinot/sql/parsers/CalciteSqlCompilerTest.java | 6 +++---
.../integration/tests/OfflineClusterIntegrationTest.java | 14 ++
3 files changed, 20 insertions(+), 3 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
index 45b3274..2aec8ec 100644
---
a/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
+++
b/pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
@@ -676,6 +676,9 @@ public class CalciteSqlParser {
try {
FunctionInvoker invoker = new FunctionInvoker(functionInfo);
Object result = invoker.process(arguments);
+if (result instanceof String) {
+ result = String.format("'%s'", result);
+}
return RequestUtils.getLiteralExpression(result);
} catch (Exception e) {
throw new SqlCompilationException(new
IllegalArgumentException("Unsupported function - " + funcName, e));
diff --git
a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
index ac0a5cf..aec7917 100644
---
a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
+++
b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
@@ -1534,7 +1534,7 @@ public class CalciteSqlCompilerTest {
Function greaterThan = pinotQuery.getFilterExpression().getFunctionCall();
String today =
greaterThan.getOperands().get(1).getLiteral().getStringValue();
String expectedTodayStr =
-
Instant.now().atZone(ZoneId.of("UTC")).format(DateTimeFormatter.ofPattern("-MM-dd
z"));
+"'" +
Instant.now().atZone(ZoneId.of("UTC")).format(DateTimeFormatter.ofPattern("-MM-dd
z")) + "'";
Assert.assertEquals(today, expectedTodayStr);
}
@@ -1557,7 +1557,7 @@ public class CalciteSqlCompilerTest {
Assert.assertTrue(expression.getLiteral() != null);
String today = expression.getLiteral().getStringValue();
String expectedTodayStr =
-
Instant.now().atZone(ZoneId.of("UTC")).format(DateTimeFormatter.ofPattern("-MM-dd
z"));
+"'" +
Instant.now().atZone(ZoneId.of("UTC")).format(DateTimeFormatter.ofPattern("-MM-dd
z")) + "'";
Assert.assertEquals(today, expectedTodayStr);
expression =
CalciteSqlParser.compileToExpression("toDateTime(playerName)");
Assert.assertTrue(expression.getFunctionCall() != null);
@@ -1575,7 +1575,7 @@ public class CalciteSqlCompilerTest {
Assert.assertTrue(expression.getFunctionCall() != null);
expression =
CalciteSqlParser.invokeCompileTimeFunctionExpression(expression);
Assert.assertTrue(expression.getLiteral() != null);
-Assert.assertEquals(expression.getLiteral().getFieldValue(), "emaNreyalp");
+Assert.assertEquals(expression.getLiteral().getFieldValue(),
"'emaNreyalp'");
expression = CalciteSqlParser.compileToExpression("count(*)");
Assert.assertTrue(expression.getFunctionCall() != null);
expression =
CalciteSqlParser.invokeCompileTimeFunctionExpression(expression);
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
index f433a3e..223602c 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
@@ -24,6 +24,9 @@ import com.fasterxml.jackson.databind.node.ObjectNode;
import com.google.common.collect.ImmutableList;
import java.io.File;
import java.io.IOException;
+import java.time.Instant;
+import java.time.ZoneId;
+import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
@@ -309,6 +312,17 @@ public class OfflineClusterIntegrationTest extends
BaseClusterIntegrationTestSet
}
@Test
+ public void testTimeFunc()
+ throws Exception {
+String sqlQuery = "SELECT toDateTime(now(), '-MM-dd z') FROM mytable";
+JsonNode response = postSqlQuery(sqlQuery, _brokerBaseApiUrl);
+String todayStr =
[GitHub] [incubator-pinot] npawar opened a new pull request #5455: Use java.io.tmpdir in KafkaStarterUtils
npawar opened a new pull request #5455: URL: https://github.com/apache/incubator-pinot/pull/5455 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [TE] add mock events loader to pinot-thirdeye docker defaults (#5454)
This is an automated email from the ASF dual-hosted git repository. apucher pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new cee2b23 [TE] add mock events loader to pinot-thirdeye docker defaults (#5454) cee2b23 is described below commit cee2b2392152839234954051fc6ce5bbfd88 Author: Alexander Pucher AuthorDate: Wed May 27 13:29:51 2020 -0700 [TE] add mock events loader to pinot-thirdeye docker defaults (#5454) --- .../pinot-thirdeye/config/ephemeral/detector.yml | 43 ++ .../images/pinot-thirdeye/config/ephemeral/rca.yml | 26 ++--- .../config/pinot-quickstart/detector.yml | 43 ++ .../pinot-thirdeye/config/pinot-quickstart/rca.yml | 26 ++--- 4 files changed, 128 insertions(+), 10 deletions(-) diff --git a/docker/images/pinot-thirdeye/config/ephemeral/detector.yml b/docker/images/pinot-thirdeye/config/ephemeral/detector.yml index 19372ff..da08eb1 100644 --- a/docker/images/pinot-thirdeye/config/ephemeral/detector.yml +++ b/docker/images/pinot-thirdeye/config/ephemeral/detector.yml @@ -17,6 +17,7 @@ alert: false autoload: false classifier: false holidayEventsLoader: false +mockEventsLoader: true monitor: false pinotProxy: false scheduler: false @@ -82,3 +83,45 @@ holidayEventsLoaderConfiguration: - "en.vietnamese#holi...@group.v.calendar.google.com" holidayLoadRange: 259200 runFrequency: 7 +mockEventsLoaderConfiguration: + generators: +- type: HOLIDAY + arrivalType: exponential + arrivalMean: 8640 + durationType: fixed + durationMean: 8640 + seed: 0 + namePrefixes: [First, Second, Third, Last, Funky, Happy, Sad, Glorious, Jolly, Unity, Pinot's] + nameSuffixes: [day, day, days, celebration, rememberance, occurrence, moment] +- type: INFORMED + arrivalType: exponential + arrivalMean: 4320 + durationType: exponential + durationMean: 360 + seed: 1 + namePrefixes: [Login, Web, Search, Catalog, Integration, Network, Backup, Ingress, Proxy, Failure, Pinot, ThirdEye] + nameSuffixes: [backend, frontend, v1.1, v1.2, v1.3, v2.0, v3, v4, v5, storage, topic, container, database] +- type: CM + arrivalType: exponential + arrivalMean: 2160 + durationType: fixed + durationMean: 180 + seed: 2 + namePrefixes: [Database, Web, Search, Catalog, Integration, Network, Backup, Ingress, Proxy, Failure, Pinot, ThirdEye] +- type: CUSTOM + arrivalType: exponential + arrivalMean: 43200 + durationType: exponential + durationMean: 8640 + seed: 3 + namePrefixes: [Marketing, Onboarding, Vaction, Outreach, InDay] + nameSuffixes: [integration, campaign, meeting] +- type: LIX + arrivalType: exponential + arrivalMean: 25920 + durationType: exponential + durationMean: 60480 + seed: 4 + namePrefixes: [System, Model, Campaign, Welcome, Pinot, ThirdEye] + nameSuffixes: [tuning, bugfix, rollout, test] + diff --git a/docker/images/pinot-thirdeye/config/ephemeral/rca.yml b/docker/images/pinot-thirdeye/config/ephemeral/rca.yml index 0c3cb2d..aecfd76 100644 --- a/docker/images/pinot-thirdeye/config/ephemeral/rca.yml +++ b/docker/images/pinot-thirdeye/config/ephemeral/rca.yml @@ -31,7 +31,11 @@ frameworks: eventExperiment: - outputName: OUTPUT inputNames: [INPUT] - className: org.apache.pinot.thirdeye.rootcause.impl.EmptyPipeline + className: org.apache.pinot.thirdeye.rootcause.impl.ThirdEyeEventsPipeline + properties: +strategy: COMPOUND +k: 500 +eventType: LIX eventHoliday: - outputName: METRIC_RELATED @@ -44,7 +48,7 @@ frameworks: properties: strategy: COMPOUND k: 500 - eventType: holiday + eventType: HOLIDAY eventCustom: - outputName: OUTPUT @@ -53,7 +57,7 @@ frameworks: properties: strategy: COMPOUND k: 500 - eventType: custom + eventType: CUSTOM eventAnomaly: - outputName: OUTPUT @@ -73,10 +77,22 @@ frameworks: eventChange: - outputName: OUTPUT inputNames: [INPUT] - className: org.apache.pinot.thirdeye.rootcause.impl.EmptyPipeline + className: org.apache.pinot.thirdeye.rootcause.impl.ThirdEyeEventsPipeline + properties: +strategy: COMPOUND +k: 500 +eventType: CM eventDeployment: - outputName: OUTPUT inputNames: [INPUT] - className: org.apache.pinot.thirdeye.rootcause.impl.EmptyPipeline + className: org.apache.pinot.thirdeye.rootcause.impl.ThirdEyeEventsPipeline +
[GitHub] [incubator-pinot] apucher merged pull request #5454: [TE] add mock events loader to pinot-thirdeye docker defaults
apucher merged pull request #5454: URL: https://github.com/apache/incubator-pinot/pull/5454 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adding-pr-template updated (ebfad8f -> 7b2b0a8)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch adding-pr-template in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from ebfad8f Update PULL_REQUEST_TEMPLATE.md add 7b2b0a8 Update PULL_REQUEST_TEMPLATE.md No new revisions were added by this update. Summary of changes: .github/PULL_REQUEST_TEMPLATE.md | 4 1 file changed, 4 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] apucher opened a new pull request #5454: [TE] add mock events loader to pinot-thirdeye docker defaults
apucher opened a new pull request #5454: URL: https://github.com/apache/incubator-pinot/pull/5454 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch mock-events-for-pinot-thirdeye-docker created (now b218d32)
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch mock-events-for-pinot-thirdeye-docker in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at b218d32 [TE] add mock events loader to pinot-thirdeye docker defaults This branch includes the following new commits: new b218d32 [TE] add mock events loader to pinot-thirdeye docker defaults The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: [TE] add mock events loader to pinot-thirdeye docker defaults
This is an automated email from the ASF dual-hosted git repository. apucher pushed a commit to branch mock-events-for-pinot-thirdeye-docker in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit b218d32bd4e9731d90c2abfd71e1e0331e2f08cd Author: Alexander Pucher AuthorDate: Wed May 27 12:27:06 2020 -0700 [TE] add mock events loader to pinot-thirdeye docker defaults --- .../pinot-thirdeye/config/ephemeral/detector.yml | 43 ++ .../images/pinot-thirdeye/config/ephemeral/rca.yml | 26 ++--- .../config/pinot-quickstart/detector.yml | 43 ++ .../pinot-thirdeye/config/pinot-quickstart/rca.yml | 26 ++--- 4 files changed, 128 insertions(+), 10 deletions(-) diff --git a/docker/images/pinot-thirdeye/config/ephemeral/detector.yml b/docker/images/pinot-thirdeye/config/ephemeral/detector.yml index 19372ff..da08eb1 100644 --- a/docker/images/pinot-thirdeye/config/ephemeral/detector.yml +++ b/docker/images/pinot-thirdeye/config/ephemeral/detector.yml @@ -17,6 +17,7 @@ alert: false autoload: false classifier: false holidayEventsLoader: false +mockEventsLoader: true monitor: false pinotProxy: false scheduler: false @@ -82,3 +83,45 @@ holidayEventsLoaderConfiguration: - "en.vietnamese#holi...@group.v.calendar.google.com" holidayLoadRange: 259200 runFrequency: 7 +mockEventsLoaderConfiguration: + generators: +- type: HOLIDAY + arrivalType: exponential + arrivalMean: 8640 + durationType: fixed + durationMean: 8640 + seed: 0 + namePrefixes: [First, Second, Third, Last, Funky, Happy, Sad, Glorious, Jolly, Unity, Pinot's] + nameSuffixes: [day, day, days, celebration, rememberance, occurrence, moment] +- type: INFORMED + arrivalType: exponential + arrivalMean: 4320 + durationType: exponential + durationMean: 360 + seed: 1 + namePrefixes: [Login, Web, Search, Catalog, Integration, Network, Backup, Ingress, Proxy, Failure, Pinot, ThirdEye] + nameSuffixes: [backend, frontend, v1.1, v1.2, v1.3, v2.0, v3, v4, v5, storage, topic, container, database] +- type: CM + arrivalType: exponential + arrivalMean: 2160 + durationType: fixed + durationMean: 180 + seed: 2 + namePrefixes: [Database, Web, Search, Catalog, Integration, Network, Backup, Ingress, Proxy, Failure, Pinot, ThirdEye] +- type: CUSTOM + arrivalType: exponential + arrivalMean: 43200 + durationType: exponential + durationMean: 8640 + seed: 3 + namePrefixes: [Marketing, Onboarding, Vaction, Outreach, InDay] + nameSuffixes: [integration, campaign, meeting] +- type: LIX + arrivalType: exponential + arrivalMean: 25920 + durationType: exponential + durationMean: 60480 + seed: 4 + namePrefixes: [System, Model, Campaign, Welcome, Pinot, ThirdEye] + nameSuffixes: [tuning, bugfix, rollout, test] + diff --git a/docker/images/pinot-thirdeye/config/ephemeral/rca.yml b/docker/images/pinot-thirdeye/config/ephemeral/rca.yml index 0c3cb2d..aecfd76 100644 --- a/docker/images/pinot-thirdeye/config/ephemeral/rca.yml +++ b/docker/images/pinot-thirdeye/config/ephemeral/rca.yml @@ -31,7 +31,11 @@ frameworks: eventExperiment: - outputName: OUTPUT inputNames: [INPUT] - className: org.apache.pinot.thirdeye.rootcause.impl.EmptyPipeline + className: org.apache.pinot.thirdeye.rootcause.impl.ThirdEyeEventsPipeline + properties: +strategy: COMPOUND +k: 500 +eventType: LIX eventHoliday: - outputName: METRIC_RELATED @@ -44,7 +48,7 @@ frameworks: properties: strategy: COMPOUND k: 500 - eventType: holiday + eventType: HOLIDAY eventCustom: - outputName: OUTPUT @@ -53,7 +57,7 @@ frameworks: properties: strategy: COMPOUND k: 500 - eventType: custom + eventType: CUSTOM eventAnomaly: - outputName: OUTPUT @@ -73,10 +77,22 @@ frameworks: eventChange: - outputName: OUTPUT inputNames: [INPUT] - className: org.apache.pinot.thirdeye.rootcause.impl.EmptyPipeline + className: org.apache.pinot.thirdeye.rootcause.impl.ThirdEyeEventsPipeline + properties: +strategy: COMPOUND +k: 500 +eventType: CM eventDeployment: - outputName: OUTPUT inputNames: [INPUT] - className: org.apache.pinot.thirdeye.rootcause.impl.EmptyPipeline + className: org.apache.pinot.thirdeye.rootcause.impl.ThirdEyeEventsPipeline + properties: +strategy: COMPOUND +k: 500 +eventType: INFORMED +eventAC: + - outputName: OUTPUT +inputNames: [INPUT] +
[GitHub] [incubator-pinot] npawar closed issue #5269: Add support for protobuf input format
npawar closed issue #5269: URL: https://github.com/apache/incubator-pinot/issues/5269 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Adding support for Protobuf input format (#5293)
This is an automated email from the ASF dual-hosted git repository.
nehapawar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new b69453e Adding support for Protobuf input format (#5293)
b69453e is described below
commit b69453e272b09314071183fc40a6516d64b0a412
Author: Kartik Khare
AuthorDate: Wed May 27 23:50:14 2020 +0530
Adding support for Protobuf input format (#5293)
Authored-by: Kartik Khare
---
.../v0_deprecated/pinot-hadoop/pom.xml | 11 +
.../hadoop/job/mappers/SegmentCreationMapper.java | 10 +
.../v0_deprecated/pinot-spark/pom.xml | 12 +-
.../spark/jobs/SparkSegmentCreationFunction.java | 10 +
.../{ => pinot-protobuf}/pom.xml | 45 +-
.../protobuf/ProtoBufRecordExtractor.java | 49 +
.../inputformat/protobuf/ProtoBufRecordReader.java | 137 +++
.../protobuf/ProtoBufRecordReaderConfig.java | 18 +-
.../protobuf/ProtoBufRecordReaderTest.java | 173
.../pinot/plugin/inputformat/protobuf/Sample.java | 1075
.../pinot-protobuf/src/test/resources/sample.desc |8 +
.../pinot-protobuf/src/test/resources/sample.proto |9 +
pinot-plugins/pinot-input-format/pom.xml |1 +
.../apache/pinot/spi/data/readers/FileFormat.java |2 +-
.../org/apache/pinot/spi/utils/ResourceFinder.java | 78 ++
15 files changed, 1600 insertions(+), 38 deletions(-)
diff --git
a/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/pom.xml
b/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/pom.xml
index 6b7450a..1fd7033 100644
--- a/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/pom.xml
+++ b/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/pom.xml
@@ -147,6 +147,17 @@
${project.version}
+ org.apache.pinot
+ pinot-protobuf
+ ${project.version}
+
+
+ com.google.protobuf
+ protobuf-java
+
+
+
+
org.apache.hadoop
hadoop-client
diff --git
a/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentCreationMapper.java
b/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentCreationMapper.java
index 47242e5..52c3393 100644
---
a/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentCreationMapper.java
+++
b/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentCreationMapper.java
@@ -44,6 +44,7 @@ import
org.apache.pinot.core.segment.name.SegmentNameGenerator;
import org.apache.pinot.core.segment.name.SimpleSegmentNameGenerator;
import org.apache.pinot.ingestion.common.JobConfigConstants;
import org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig;
+import org.apache.pinot.plugin.inputformat.protobuf.ProtoBufRecordReaderConfig;
import org.apache.pinot.plugin.inputformat.thrift.ThriftRecordReaderConfig;
import org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig;
import org.apache.pinot.spi.config.table.TableConfig;
@@ -330,6 +331,15 @@ public class SegmentCreationMapper extends
Mapper
-
+
+ org.apache.pinot
+ pinot-protobuf
+ ${project.version}
+
+
+ com.google.protobuf
+ protobuf-java
+
+
+
org.apache.spark
diff --git
a/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/src/main/java/org/apache/pinot/spark/jobs/SparkSegmentCreationFunction.java
b/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/src/main/java/org/apache/pinot/spark/jobs/SparkSegmentCreationFunction.java
index 80d7434..b234e3e 100644
---
a/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/src/main/java/org/apache/pinot/spark/jobs/SparkSegmentCreationFunction.java
+++
b/pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/src/main/java/org/apache/pinot/spark/jobs/SparkSegmentCreationFunction.java
@@ -41,6 +41,7 @@ import
org.apache.pinot.core.segment.name.SegmentNameGenerator;
import org.apache.pinot.core.segment.name.SimpleSegmentNameGenerator;
import org.apache.pinot.ingestion.common.JobConfigConstants;
import org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig;
+import org.apache.pinot.plugin.inputformat.protobuf.ProtoBufRecordReaderConfig;
import org.apache.pinot.plugin.inputformat.thrift.ThriftRecordReaderConfig;
import org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig;
import org.apache.pinot.spi.config.table.TableConfig;
@@ -297,6 +298,15 @@ public class SparkSegmentCreationFunction implements
Serializable {
return
[GitHub] [incubator-pinot] npawar merged pull request #5293: Adding support for Protobuf input format
npawar merged pull request #5293: URL: https://github.com/apache/incubator-pinot/pull/5293 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on pull request #5293: Adding support for Protobuf input format
npawar commented on pull request #5293: URL: https://github.com/apache/incubator-pinot/pull/5293#issuecomment-634850943 Linking to issue https://github.com/apache/incubator-pinot/issues/5269 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (008be2d -> 3514a17)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from 008be2d Make Literal transformer return string literals (#5453)
add 3514a17 Add PinotServiceManager to start Pinot components (#5266)
No new revisions were added by this update.
Summary of changes:
pinot-broker/pom.xml | 1 -
.../broker/broker/helix/HelixBrokerStarter.java| 32 ++-
.../broker/broker/HelixBrokerStarterTest.java | 2 +-
.../pinot/common/metrics/AbstractMetrics.java | 4 +
.../apache/pinot/common/utils/CommonConstants.java | 4 +
.../apache/pinot/common/utils/ServiceStatus.java | 144
.../apache/pinot/controller/ControllerStarter.java | 28 ++-
.../pinot/integration/tests/ClusterTest.java | 6 +-
.../tests/ServerStarterIntegrationTest.java| 1 +
.../org/apache/pinot/minion/MinionStarter.java | 25 ++-
.../pinot/server/starter/ServerInstance.java | 3 +-
.../server/starter/helix/HelixServerStarter.java | 213 ++
.../org/apache/pinot/spi/services/ServiceRole.java | 9 +-
.../pinot/spi/services/ServiceStartable.java | 26 ++-
pinot-tools/pom.xml| 11 +
.../pinot/tools/admin/PinotAdministrator.java | 2 +
...Server.java => PinotServiceManagerStarter.java} | 7 +-
.../admin/command/AbstractBaseAdminCommand.java| 31 +--
.../admin/command/EnumArrayOptionHandler.java | 67 ++
.../tools/admin/command/StartBrokerCommand.java| 45 ++--
.../admin/command/StartControllerCommand.java | 117 +++---
.../tools/admin/command/StartServerCommand.java| 57 ++---
.../admin/command/StartServiceManagerCommand.java | 213 ++
.../pinot/tools/perf/PerfBenchmarkDriver.java | 3 +-
.../pinot/tools/service/PinotServiceManager.java | 229
.../PinotServiceManagerAdminApiApplication.java| 28 +--
.../service/PinotServiceManagerStatusCallback.java | 39 ++--
.../service/api/resources/PinotInstanceStatus.java | 59 +
.../resources/PinotServiceManagerHealthCheck.java | 79 +++
.../PinotServiceManagerInstanceResource.java | 241 +
.../apache/pinot/tools/utils/PinotConfigUtils.java | 189
...ckstart-log4j2.xml => pinot-service-log4j2.xml} | 10 +-
pom.xml| 6 +
33 files changed, 1555 insertions(+), 376 deletions(-)
copy
pinot-core/src/main/java/org/apache/pinot/core/segment/creator/ForwardIndexType.java
=> pinot-spi/src/main/java/org/apache/pinot/spi/services/ServiceRole.java (82%)
copy
pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcher.java
=> pinot-spi/src/main/java/org/apache/pinot/spi/services/ServiceStartable.java
(71%)
copy pinot-tools/src/main/java/org/apache/pinot/tools/admin/{PinotServer.java
=> PinotServiceManagerStarter.java} (87%)
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/EnumArrayOptionHandler.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/StartServiceManagerCommand.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/service/PinotServiceManager.java
copy
pinot-broker/src/main/java/org/apache/pinot/broker/broker/BrokerAdminApiApplication.java
=>
pinot-tools/src/main/java/org/apache/pinot/tools/service/PinotServiceManagerAdminApiApplication.java
(74%)
copy
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SizeBasedDocIdIterator.java
=>
pinot-tools/src/main/java/org/apache/pinot/tools/service/PinotServiceManagerStatusCallback.java
(50%)
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/service/api/resources/PinotInstanceStatus.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/service/api/resources/PinotServiceManagerHealthCheck.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/service/api/resources/PinotServiceManagerInstanceResource.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/utils/PinotConfigUtils.java
copy pinot-tools/src/main/resources/conf/{quickstart-log4j2.xml =>
pinot-service-log4j2.xml} (88%)
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (008be2d -> 3514a17)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from 008be2d Make Literal transformer return string literals (#5453)
add 3514a17 Add PinotServiceManager to start Pinot components (#5266)
No new revisions were added by this update.
Summary of changes:
pinot-broker/pom.xml | 1 -
.../broker/broker/helix/HelixBrokerStarter.java| 32 ++-
.../broker/broker/HelixBrokerStarterTest.java | 2 +-
.../pinot/common/metrics/AbstractMetrics.java | 4 +
.../apache/pinot/common/utils/CommonConstants.java | 4 +
.../apache/pinot/common/utils/ServiceStatus.java | 144
.../apache/pinot/controller/ControllerStarter.java | 28 ++-
.../pinot/integration/tests/ClusterTest.java | 6 +-
.../tests/ServerStarterIntegrationTest.java| 1 +
.../org/apache/pinot/minion/MinionStarter.java | 25 ++-
.../pinot/server/starter/ServerInstance.java | 3 +-
.../server/starter/helix/HelixServerStarter.java | 213 ++
.../org/apache/pinot/spi/services/ServiceRole.java | 9 +-
.../pinot/spi/services/ServiceStartable.java | 26 ++-
pinot-tools/pom.xml| 11 +
.../pinot/tools/admin/PinotAdministrator.java | 2 +
...Server.java => PinotServiceManagerStarter.java} | 7 +-
.../admin/command/AbstractBaseAdminCommand.java| 31 +--
.../admin/command/EnumArrayOptionHandler.java | 67 ++
.../tools/admin/command/StartBrokerCommand.java| 45 ++--
.../admin/command/StartControllerCommand.java | 117 +++---
.../tools/admin/command/StartServerCommand.java| 57 ++---
.../admin/command/StartServiceManagerCommand.java | 213 ++
.../pinot/tools/perf/PerfBenchmarkDriver.java | 3 +-
.../pinot/tools/service/PinotServiceManager.java | 229
.../PinotServiceManagerAdminApiApplication.java| 28 +--
.../service/PinotServiceManagerStatusCallback.java | 39 ++--
.../service/api/resources/PinotInstanceStatus.java | 59 +
.../resources/PinotServiceManagerHealthCheck.java | 79 +++
.../PinotServiceManagerInstanceResource.java | 241 +
.../apache/pinot/tools/utils/PinotConfigUtils.java | 189
...ckstart-log4j2.xml => pinot-service-log4j2.xml} | 10 +-
pom.xml| 6 +
33 files changed, 1555 insertions(+), 376 deletions(-)
copy
pinot-core/src/main/java/org/apache/pinot/core/segment/creator/ForwardIndexType.java
=> pinot-spi/src/main/java/org/apache/pinot/spi/services/ServiceRole.java (82%)
copy
pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcher.java
=> pinot-spi/src/main/java/org/apache/pinot/spi/services/ServiceStartable.java
(71%)
copy pinot-tools/src/main/java/org/apache/pinot/tools/admin/{PinotServer.java
=> PinotServiceManagerStarter.java} (87%)
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/EnumArrayOptionHandler.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/StartServiceManagerCommand.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/service/PinotServiceManager.java
copy
pinot-broker/src/main/java/org/apache/pinot/broker/broker/BrokerAdminApiApplication.java
=>
pinot-tools/src/main/java/org/apache/pinot/tools/service/PinotServiceManagerAdminApiApplication.java
(74%)
copy
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SizeBasedDocIdIterator.java
=>
pinot-tools/src/main/java/org/apache/pinot/tools/service/PinotServiceManagerStatusCallback.java
(50%)
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/service/api/resources/PinotInstanceStatus.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/service/api/resources/PinotServiceManagerHealthCheck.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/service/api/resources/PinotServiceManagerInstanceResource.java
create mode 100644
pinot-tools/src/main/java/org/apache/pinot/tools/utils/PinotConfigUtils.java
copy pinot-tools/src/main/resources/conf/{quickstart-log4j2.xml =>
pinot-service-log4j2.xml} (88%)
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on pull request #5266: Add PinotServiceManager to start Pinot components
fx19880617 commented on pull request #5266: URL: https://github.com/apache/incubator-pinot/pull/5266#issuecomment-634823281 > We will fix the incompat at our end. If for some reason we need some changes, we will come back with those. thanks Thanks for taking time to review on this ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 merged pull request #5266: Add PinotServiceManager to start Pinot components
fx19880617 merged pull request #5266: URL: https://github.com/apache/incubator-pinot/pull/5266 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5293: Adding support for Protobuf input format
KKcorps commented on a change in pull request #5293:
URL: https://github.com/apache/incubator-pinot/pull/5293#discussion_r431309682
##
File path:
pinot-plugins/pinot-input-format/pinot-protobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordReader.java
##
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.protobuf;
+
+import com.google.protobuf.DescriptorProtos;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.data.readers.RecordReaderUtils;
+import org.apache.pinot.spi.utils.ResourceFinder;
+import org.apache.pinot.spi.utils.SchemaFieldExtractorUtils;
+
+
+public class ProtoBufRecordReader implements RecordReader {
+ private File _dataFile;
+ private Schema _schema;
+ private ProtoBufRecordExtractor _recordExtractor;
+
+ private InputStream _inputStream;
+ private boolean _hasNext;
+ private Descriptors.Descriptor _descriptor;
+
+ private boolean hasMoreToRead()
+ throws IOException {
+_inputStream.mark(1);
+int nextByte = _inputStream.read();
+_inputStream.reset();
+return nextByte != -1;
+ }
+
+ private void init()
+ throws IOException {
+_inputStream = RecordReaderUtils.getBufferedInputStream(_dataFile);
+try {
+ _hasNext = hasMoreToRead();
+} catch (Exception e) {
+ _inputStream.close();
+ throw e;
+}
+ }
+
+ @Override
+ public void init(File dataFile, Schema schema, @Nullable RecordReaderConfig
recordReaderConfig)
+ throws IOException {
+_dataFile = dataFile;
+_schema = schema;
+Set sourceFields = SchemaFieldExtractorUtils.extract(schema);
+ProtoBufRecordExtractorConfig recordExtractorConfig = new
ProtoBufRecordExtractorConfig();
+ProtoBufRecordReaderConfig protoBufRecordReaderConfig =
(ProtoBufRecordReaderConfig) recordReaderConfig;
+InputStream fin = getDescriptorFileInputStream(protoBufRecordReaderConfig);
+buildProtoBufDescriptor(fin);
+_recordExtractor = new ProtoBufRecordExtractor();
+_recordExtractor.init(sourceFields, recordExtractorConfig);
+init();
+ }
+
+ private void buildProtoBufDescriptor(InputStream fin)
+ throws IOException {
+try {
+ DescriptorProtos.FileDescriptorSet set =
DescriptorProtos.FileDescriptorSet.parseFrom(fin);
+ Descriptors.FileDescriptor fileDescriptor =
+ Descriptors.FileDescriptor.buildFrom(set.getFile(0), new
Descriptors.FileDescriptor[]{});
+ _descriptor = fileDescriptor.getMessageTypes().get(0);
+} catch (Descriptors.DescriptorValidationException e) {
+ throw new IOException("Descriptor file validation failed", e);
+}
+ }
+
+ private InputStream getDescriptorFileInputStream(ProtoBufRecordReaderConfig
protoBufRecordReaderConfig)
+ throws IOException {
+URI descriptorFileURI = protoBufRecordReaderConfig.getDescriptorFile();
+return ResourceFinder.openResource(descriptorFileURI);
+ }
+
+ @Override
+ public boolean hasNext() {
+return _hasNext;
+ }
+
+ @Override
+ public GenericRow next()
+ throws IOException {
+return next(new GenericRow());
+ }
+
+ @Override
+ public GenericRow next(GenericRow reuse)
+ throws IOException {
+Message message = null;
+try {
+ DynamicMessage tmp = DynamicMessage.getDefaultInstance(_descriptor);
+ Message.Builder builder = tmp.newBuilderForType();
Review comment:
Done. Temp DynamicMessage can be reused but reusing builder is not
possible.
Moved the temp Dyanmic message to class variable.
This is an automated message from the Apache Git Service.
To respond to the
[GitHub] [incubator-pinot] jamesyfshao commented on pull request #5394: add callback interface for upsert component
jamesyfshao commented on pull request #5394: URL: https://github.com/apache/incubator-pinot/pull/5394#issuecomment-634800807 @kishoreg @mcvsubbu @Jackie-Jiang appreciated if you guys can take a look at the diff this week when you have time This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #5293: Adding support for Protobuf input format
npawar commented on a change in pull request #5293:
URL: https://github.com/apache/incubator-pinot/pull/5293#discussion_r431285868
##
File path:
pinot-plugins/pinot-input-format/pinot-protobuf/src/main/java/org/apache/pinot/plugin/inputformat/protobuf/ProtoBufRecordReader.java
##
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.inputformat.protobuf;
+
+import com.google.protobuf.DescriptorProtos;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import com.google.protobuf.Message;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.data.readers.RecordReaderConfig;
+import org.apache.pinot.spi.data.readers.RecordReaderUtils;
+import org.apache.pinot.spi.utils.ResourceFinder;
+import org.apache.pinot.spi.utils.SchemaFieldExtractorUtils;
+
+
+public class ProtoBufRecordReader implements RecordReader {
+ private File _dataFile;
+ private Schema _schema;
+ private ProtoBufRecordExtractor _recordExtractor;
+
+ private InputStream _inputStream;
+ private boolean _hasNext;
+ private Descriptors.Descriptor _descriptor;
+
+ private boolean hasMoreToRead()
+ throws IOException {
+_inputStream.mark(1);
+int nextByte = _inputStream.read();
+_inputStream.reset();
+return nextByte != -1;
+ }
+
+ private void init()
+ throws IOException {
+_inputStream = RecordReaderUtils.getBufferedInputStream(_dataFile);
+try {
+ _hasNext = hasMoreToRead();
+} catch (Exception e) {
+ _inputStream.close();
+ throw e;
+}
+ }
+
+ @Override
+ public void init(File dataFile, Schema schema, @Nullable RecordReaderConfig
recordReaderConfig)
+ throws IOException {
+_dataFile = dataFile;
+_schema = schema;
+Set sourceFields = SchemaFieldExtractorUtils.extract(schema);
+ProtoBufRecordExtractorConfig recordExtractorConfig = new
ProtoBufRecordExtractorConfig();
+ProtoBufRecordReaderConfig protoBufRecordReaderConfig =
(ProtoBufRecordReaderConfig) recordReaderConfig;
+InputStream fin = getDescriptorFileInputStream(protoBufRecordReaderConfig);
+buildProtoBufDescriptor(fin);
+_recordExtractor = new ProtoBufRecordExtractor();
+_recordExtractor.init(sourceFields, recordExtractorConfig);
+init();
+ }
+
+ private void buildProtoBufDescriptor(InputStream fin)
+ throws IOException {
+try {
+ DescriptorProtos.FileDescriptorSet set =
DescriptorProtos.FileDescriptorSet.parseFrom(fin);
+ Descriptors.FileDescriptor fileDescriptor =
+ Descriptors.FileDescriptor.buildFrom(set.getFile(0), new
Descriptors.FileDescriptor[]{});
+ _descriptor = fileDescriptor.getMessageTypes().get(0);
+} catch (Descriptors.DescriptorValidationException e) {
+ throw new IOException("Descriptor file validation failed", e);
+}
+ }
+
+ private InputStream getDescriptorFileInputStream(ProtoBufRecordReaderConfig
protoBufRecordReaderConfig)
+ throws IOException {
+URI descriptorFileURI = protoBufRecordReaderConfig.getDescriptorFile();
+return ResourceFinder.openResource(descriptorFileURI);
+ }
+
+ @Override
+ public boolean hasNext() {
+return _hasNext;
+ }
+
+ @Override
+ public GenericRow next()
+ throws IOException {
+return next(new GenericRow());
+ }
+
+ @Override
+ public GenericRow next(GenericRow reuse)
+ throws IOException {
+Message message = null;
+try {
+ DynamicMessage tmp = DynamicMessage.getDefaultInstance(_descriptor);
+ Message.Builder builder = tmp.newBuilderForType();
Review comment:
can you check with the help of your test? If they can be reused, make
them class fields?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
[GitHub] [incubator-pinot] fx19880617 merged pull request #5453: Make Literal transformer return string literals
fx19880617 merged pull request #5453: URL: https://github.com/apache/incubator-pinot/pull/5453 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Make Literal transformer return string literals (#5453)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 008be2d Make Literal transformer return string literals (#5453)
008be2d is described below
commit 008be2db874dd1c0d7877ce712842abd818d89d1
Author: Xiang Fu
AuthorDate: Wed May 27 02:47:03 2020 -0700
Make Literal transformer return string literals (#5453)
---
.../pql/parsers/PinotQuery2BrokerRequestConverter.java | 2 +-
.../apache/pinot/sql/parsers/CalciteSqlCompilerTest.java| 13 +
.../transform/function/LiteralTransformFunction.java| 12 +---
3 files changed, 23 insertions(+), 4 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java
b/pinot-common/src/main/java/org/apache/pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java
index 1ce9a06..f4a9639 100644
---
a/pinot-common/src/main/java/org/apache/pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java
+++
b/pinot-common/src/main/java/org/apache/pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java
@@ -128,7 +128,7 @@ public class PinotQuery2BrokerRequestConverter {
if (selection == null) {
selection = new Selection();
}
-
selection.addToSelectionColumns(expression.getLiteral().getStringValue());
+
selection.addToSelectionColumns(expression.getLiteral().getFieldValue().toString());
break;
case IDENTIFIER:
if (selection == null) {
diff --git
a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
index a037b3f..ac0a5cf 100644
---
a/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
+++
b/pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
@@ -36,6 +36,7 @@ import org.apache.pinot.common.request.ExpressionType;
import org.apache.pinot.common.request.FilterOperator;
import org.apache.pinot.common.request.Function;
import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Literal;
import org.apache.pinot.common.request.PinotQuery;
import org.apache.pinot.pql.parsers.PinotQuery2BrokerRequestConverter;
import org.apache.pinot.pql.parsers.Pql2Compiler;
@@ -293,6 +294,18 @@ public class CalciteSqlCompilerTest {
}
@Test
+ public void testBrokerConverterWithLiteral() {
+PinotQuery pinotQuery = CalciteSqlParser.compileToPinotQuery("select now()
from mytable");
+Literal literal = pinotQuery.getSelectList().get(0).getLiteral();
+Assert.assertNotNull(literal);
+PinotQuery2BrokerRequestConverter converter = new
PinotQuery2BrokerRequestConverter();
+BrokerRequest tempBrokerRequest = converter.convert(pinotQuery);
+Assert.assertEquals(tempBrokerRequest.getQuerySource().getTableName(),
"mytable");
+
Assert.assertEquals(tempBrokerRequest.getSelections().getSelectionColumns().get(0),
+literal.getFieldValue().toString());
+ }
+
+ @Test
public void testSelectAs() {
PinotQuery pinotQuery = CalciteSqlParser.compileToPinotQuery(
"select sum(A) as sum_A, count(B) as count_B from vegetables where g
IN (12, 13, 15.2, 17)");
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
index 24a2374..4317a2b 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
@@ -18,11 +18,13 @@
*/
package org.apache.pinot.core.operator.transform.function;
+import java.util.Arrays;
import java.util.List;
import java.util.Map;
import org.apache.pinot.core.common.DataSource;
import org.apache.pinot.core.operator.blocks.ProjectionBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
import org.apache.pinot.core.segment.index.readers.Dictionary;
@@ -32,6 +34,7 @@ import org.apache.pinot.core.segment.index.readers.Dictionary;
*/
public class LiteralTransformFunction implements TransformFunction {
private final String _literal;
+ private String[] _result;
public LiteralTransformFunction(String literal) {
_literal = literal;
@@ -48,12 +51,11 @@ public class LiteralTransformFunction implements
TransformFunction {
@Override
public void init(List arguments, Map
dataSourceMap) {
-throw new UnsupportedOperationException();
}
@Override
public TransformResultMetadata getResultMetadata() {
[incubator-pinot] annotated tag pinot-0.4.0 updated (fb62e74 -> 22452aa)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to annotated tag pinot-0.4.0 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag pinot-0.4.0 was modified! *** from fb62e74 (commit) to 22452aa (tag) tagging fb62e743d8b15890c9c9abfcb5ead5a47c732bf2 (commit) by Xiang Fu on Wed May 27 01:26:12 2020 -0700 - Log - [maven-release-plugin] copy for tag pinot-0.4.0 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch release-0.4.0-rc1 updated: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch release-0.4.0-rc1 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/release-0.4.0-rc1 by this push: new 3f81191 [maven-release-plugin] prepare for next development iteration 3f81191 is described below commit 3f8119124a7ce687678a355fb0e1443f2554b02d Author: Xiang Fu AuthorDate: Wed May 27 01:26:16 2020 -0700 [maven-release-plugin] prepare for next development iteration --- pinot-broker/pom.xml | 2 +- pinot-clients/pinot-java-client/pom.xml | 2 +- pinot-clients/pom.xml | 2 +- pinot-common/pom.xml | 2 +- pinot-controller/pom.xml | 2 +- pinot-core/pom.xml| 2 +- pinot-distribution/pom.xml| 2 +- pinot-integration-tests/pom.xml | 2 +- pinot-minion/pom.xml | 2 +- pinot-perf/pom.xml| 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-common/pom.xml| 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-hadoop/pom.xml| 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-spark/pom.xml | 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-standalone/pom.xml| 2 +- pinot-plugins/pinot-batch-ingestion/pom.xml | 2 +- .../pinot-batch-ingestion/v0_deprecated/pinot-hadoop/pom.xml | 2 +- .../v0_deprecated/pinot-ingestion-common/pom.xml | 2 +- pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/pom.xml | 2 +- pinot-plugins/pinot-batch-ingestion/v0_deprecated/pom.xml | 2 +- pinot-plugins/pinot-file-system/pinot-adls/pom.xml| 2 +- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 2 +- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml| 2 +- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 2 +- pinot-plugins/pinot-file-system/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-avro-base/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-avro/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-confluent-avro/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-csv/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-json/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-orc/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-parquet/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-thrift/pom.xml | 2 +- pinot-plugins/pinot-input-format/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pinot-kafka-0.9/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pinot-kafka-base/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pom.xml | 2 +- pinot-plugins/pom.xml | 2 +- pinot-server/pom.xml | 2 +- pinot-spi/pom.xml | 2 +- pinot-tools/pom.xml | 2 +- pom.xml | 4 ++-- 42 files changed, 43 insertions(+), 43 deletions(-) diff --git a/pinot-broker/pom.xml b/pinot-broker/pom.xml index 2fd885d..a884630 100644 --- a/pinot-broker/pom.xml +++ b/pinot-broker/pom.xml @@ -24,7 +24,7 @@ pinot org.apache.pinot -0.4.0 +0.5.0-SNAPSHOT .. pinot-broker diff --git a/pinot-clients/pinot-java-client/pom.xml b/pinot-clients/pinot-java-client/pom.xml index 615c5e9..60177d7 100644 --- a/pinot-clients/pinot-java-client/pom.xml +++ b/pinot-clients/pinot-java-client/pom.xml @@ -24,7 +24,7 @@ pinot-clients org.apache.pinot -0.4.0 +0.5.0-SNAPSHOT .. pinot-java-client diff --git a/pinot-clients/pom.xml b/pinot-clients/pom.xml index e05e7ec..902f54f 100644 --- a/pinot-clients/pom.xml +++ b/pinot-clients/pom.xml @@ -24,7 +24,7 @@ pinot org.apache.pinot -0.4.0 +0.5.0-SNAPSHOT .. pinot-clients diff --git a/pinot-common/pom.xml b/pinot-common/pom.xml index db2d28d..7f2a7fc 100644 --- a/pinot-common/pom.xml +++ b/pinot-common/pom.xml @@ -24,7 +24,7 @@ pinot org.apache.pinot -0.4.0 +0.5.0-SNAPSHOT ..
[incubator-pinot] branch release-0.4.0-rc1 updated: [maven-release-plugin] prepare release pinot-0.4.0
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch release-0.4.0-rc1
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/release-0.4.0-rc1 by this push:
new fb62e74 [maven-release-plugin] prepare release pinot-0.4.0
fb62e74 is described below
commit fb62e743d8b15890c9c9abfcb5ead5a47c732bf2
Author: Xiang Fu
AuthorDate: Wed May 27 01:25:57 2020 -0700
[maven-release-plugin] prepare release pinot-0.4.0
---
pinot-broker/pom.xml | 5 ++---
pinot-clients/pinot-java-client/pom.xml | 5 ++---
pinot-clients/pom.xml | 6 ++
pinot-common/pom.xml | 5 ++---
pinot-controller/pom.xml | 5 ++---
pinot-core/pom.xml| 5 ++---
pinot-distribution/pom.xml| 7 +++
pinot-integration-tests/pom.xml | 5 ++---
pinot-minion/pom.xml | 5 ++---
pinot-perf/pom.xml| 5 ++---
.../pinot-batch-ingestion/pinot-batch-ingestion-common/pom.xml| 6 ++
.../pinot-batch-ingestion/pinot-batch-ingestion-hadoop/pom.xml| 6 ++
.../pinot-batch-ingestion/pinot-batch-ingestion-spark/pom.xml | 6 ++
.../pinot-batch-ingestion-standalone/pom.xml | 6 ++
pinot-plugins/pinot-batch-ingestion/pom.xml | 6 ++
.../pinot-batch-ingestion/v0_deprecated/pinot-hadoop/pom.xml | 7 +++
.../v0_deprecated/pinot-ingestion-common/pom.xml | 6 ++
.../pinot-batch-ingestion/v0_deprecated/pinot-spark/pom.xml | 7 +++
pinot-plugins/pinot-batch-ingestion/v0_deprecated/pom.xml | 6 ++
pinot-plugins/pinot-file-system/pinot-adls/pom.xml| 5 ++---
pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 ++
pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml| 5 ++---
pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 6 ++
pinot-plugins/pinot-file-system/pom.xml | 6 ++
pinot-plugins/pinot-input-format/pinot-avro-base/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pinot-avro/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pinot-confluent-avro/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pinot-csv/pom.xml| 5 ++---
pinot-plugins/pinot-input-format/pinot-json/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pinot-orc/pom.xml| 6 ++
pinot-plugins/pinot-input-format/pinot-parquet/pom.xml| 5 ++---
pinot-plugins/pinot-input-format/pinot-thrift/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pom.xml | 6 ++
pinot-plugins/pinot-stream-ingestion/pinot-kafka-0.9/pom.xml | 6 ++
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 ++
pinot-plugins/pinot-stream-ingestion/pinot-kafka-base/pom.xml | 6 ++
pinot-plugins/pinot-stream-ingestion/pom.xml | 6 ++
pinot-plugins/pom.xml | 8 +++-
pinot-server/pom.xml | 5 ++---
pinot-spi/pom.xml | 5 ++---
pinot-tools/pom.xml | 5 ++---
pom.xml | 7 +++
42 files changed, 89 insertions(+), 149 deletions(-)
diff --git a/pinot-broker/pom.xml b/pinot-broker/pom.xml
index d9e48e1..2fd885d 100644
--- a/pinot-broker/pom.xml
+++ b/pinot-broker/pom.xml
@@ -19,13 +19,12 @@
under the License.
-->
-http://www.w3.org/2001/XMLSchema-instance;
xmlns="http://maven.apache.org/POM/4.0.0;
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0
pinot
org.apache.pinot
-${revision}${sha1}
+0.4.0
..
pinot-broker
diff --git a/pinot-clients/pinot-java-client/pom.xml
b/pinot-clients/pinot-java-client/pom.xml
index 6a98e3d..615c5e9 100644
--- a/pinot-clients/pinot-java-client/pom.xml
+++ b/pinot-clients/pinot-java-client/pom.xml
@@ -19,13 +19,12 @@
under the License.
-->
-http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5266: Add PinotServiceManager to start Pinot components
fx19880617 commented on a change in pull request #5266:
URL: https://github.com/apache/incubator-pinot/pull/5266#discussion_r430928817
##
File path:
pinot-common/src/main/java/org/apache/pinot/common/utils/ServiceStatus.java
##
@@ -43,17 +44,79 @@
*/
@SuppressWarnings("unused")
public class ServiceStatus {
+ public static final String STATUS_DESCRIPTION_NONE = "None";
+ public static final String STATUS_DESCRIPTION_INIT = "Init";
+ public static final String STATUS_DESCRIPTION_STARTED = "Started";
+ public static final String STATUS_DESCRIPTION_NO_HELIX_STATE = "Helix state
does not exist";
private static final Logger LOGGER =
LoggerFactory.getLogger(ServiceStatus.class);
+ private static final int MAX_RESOURCE_NAMES_TO_LOG = 5;
+ private static final Map
serviceStatusCallbackMap = new ConcurrentHashMap<>();
+ private static final ServiceStatusCallback serviceStatusCallback =
+ new
MapBasedMultipleCallbackServiceStatusCallback(serviceStatusCallbackMap);
- public enum Status {
-STARTING, GOOD, BAD
+ public static void setServiceStatusCallback(String name,
ServiceStatusCallback serviceStatusCallback) {
+ServiceStatus.serviceStatusCallbackMap.put(name, serviceStatusCallback);
}
- public static final String STATUS_DESCRIPTION_NONE = "None";
- public static final String STATUS_DESCRIPTION_INIT = "Init";
- public static final String STATUS_DESCRIPTION_NO_HELIX_STATE = "Helix state
does not exist";
+ public static void removeServiceStatusCallback(String name) {
+ServiceStatus.serviceStatusCallbackMap.remove(name);
+ }
- private static final int MAX_RESOURCE_NAMES_TO_LOG = 5;
+ public static Status getServiceStatus() {
+return getServiceStatus(serviceStatusCallback);
+ }
+
+ public static Status getServiceStatus(String name) {
+if (serviceStatusCallbackMap.containsKey(name)) {
+ return getServiceStatus(serviceStatusCallbackMap.get(name));
+} else {
+ return Status.NOT_STARTED;
Review comment:
Got it. Added `SHUTTING_DOWN` status.
My feeling is that during shutting down phase, the
`getServiceStatus(serviceStatusCallbackMap.get(name))` should return
`SHUTTING_DOWN`.
Once server is completely shutdown, it will be removed from the
`serviceStatusCallbackMap` so the status will be `NOT_STARTED`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [TE] fix dockerfile to account for #5428 changes (#5452)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new ac443f4 [TE] fix dockerfile to account for #5428 changes (#5452)
ac443f4 is described below
commit ac443f4c322066311902e6628dd54f2acd56be2e
Author: Alexander Pucher
AuthorDate: Wed May 27 00:23:58 2020 -0700
[TE] fix dockerfile to account for #5428 changes (#5452)
Co-authored-by: Alexander Pucher
---
docker/images/pinot-thirdeye/Dockerfile | 3 +++
1 file changed, 3 insertions(+)
diff --git a/docker/images/pinot-thirdeye/Dockerfile
b/docker/images/pinot-thirdeye/Dockerfile
index a6d9a73..42e23c7 100644
--- a/docker/images/pinot-thirdeye/Dockerfile
+++ b/docker/images/pinot-thirdeye/Dockerfile
@@ -50,6 +50,9 @@ RUN npm install -g phantomjs --unsafe-perm --ignore-scripts
RUN git clone ${PINOT_GIT_URL} ${TE_BUILD_DIR} \
&& cd ${TE_BUILD_DIR}/thirdeye \
&& git checkout ${PINOT_BRANCH} \
+&& cd thirdeye-frontend \
+&& mvn clean install -X -DskipTests || exit 1 \
+&& cd .. \
&& mvn clean install -X -DskipTests || exit 1 \
&& mkdir -p ${TE_HOME}/config/default \
&& mkdir -p ${TE_HOME}/bin \
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 merged pull request #5452: [TE] fix dockerfile to account for #5428 changes
fx19880617 merged pull request #5452: URL: https://github.com/apache/incubator-pinot/pull/5452 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch fixing_literal_in_selection updated (ca63e41 -> 04c437a)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch fixing_literal_in_selection in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard ca63e41 Make Literal transformer return string literals add 04c437a Make Literal transformer return string literals This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (ca63e41) \ N -- N -- N refs/heads/fixing_literal_in_selection (04c437a) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../apache/pinot/sql/parsers/CalciteSqlCompilerTest.java| 13 + 1 file changed, 13 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on pull request #5453: Make Literal transformer return string literals
kishoreg commented on pull request #5453: URL: https://github.com/apache/incubator-pinot/pull/5453#issuecomment-634469047 Add a test case please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5453: Make Literal transformer return string literals
fx19880617 opened a new pull request #5453: URL: https://github.com/apache/incubator-pinot/pull/5453 - Let LiteralTransformFunction return strings. - Convert Literal object to string for Selection List This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch fixing_literal_in_selection created (now ca63e41)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch fixing_literal_in_selection in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at ca63e41 Make Literal transformer return string literals This branch includes the following new commits: new ca63e41 Make Literal transformer return string literals The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Make Literal transformer return string literals
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch fixing_literal_in_selection
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit ca63e41d1ba9f690e12e4d98e2c6078d0b689c0e
Author: Xiang Fu
AuthorDate: Tue May 26 23:55:03 2020 -0700
Make Literal transformer return string literals
---
.../pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java | 2 +-
.../transform/function/LiteralTransformFunction.java | 12 +---
2 files changed, 10 insertions(+), 4 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java
b/pinot-common/src/main/java/org/apache/pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java
index 1ce9a06..f4a9639 100644
---
a/pinot-common/src/main/java/org/apache/pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java
+++
b/pinot-common/src/main/java/org/apache/pinot/pql/parsers/PinotQuery2BrokerRequestConverter.java
@@ -128,7 +128,7 @@ public class PinotQuery2BrokerRequestConverter {
if (selection == null) {
selection = new Selection();
}
-
selection.addToSelectionColumns(expression.getLiteral().getStringValue());
+
selection.addToSelectionColumns(expression.getLiteral().getFieldValue().toString());
break;
case IDENTIFIER:
if (selection == null) {
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
index 24a2374..4317a2b 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/LiteralTransformFunction.java
@@ -18,11 +18,13 @@
*/
package org.apache.pinot.core.operator.transform.function;
+import java.util.Arrays;
import java.util.List;
import java.util.Map;
import org.apache.pinot.core.common.DataSource;
import org.apache.pinot.core.operator.blocks.ProjectionBlock;
import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
import org.apache.pinot.core.segment.index.readers.Dictionary;
@@ -32,6 +34,7 @@ import org.apache.pinot.core.segment.index.readers.Dictionary;
*/
public class LiteralTransformFunction implements TransformFunction {
private final String _literal;
+ private String[] _result;
public LiteralTransformFunction(String literal) {
_literal = literal;
@@ -48,12 +51,11 @@ public class LiteralTransformFunction implements
TransformFunction {
@Override
public void init(List arguments, Map
dataSourceMap) {
-throw new UnsupportedOperationException();
}
@Override
public TransformResultMetadata getResultMetadata() {
-throw new UnsupportedOperationException();
+return BaseTransformFunction.STRING_SV_NO_DICTIONARY_METADATA;
}
@Override
@@ -93,7 +95,11 @@ public class LiteralTransformFunction implements
TransformFunction {
@Override
public String[] transformToStringValuesSV(ProjectionBlock projectionBlock) {
-throw new UnsupportedOperationException();
+if (_result == null) {
+ _result = new String[DocIdSetPlanNode.MAX_DOC_PER_CALL];
+ Arrays.fill(_result, _literal);
+}
+return _result;
}
@Override
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
