(pinot) branch master updated (870cf57635 -> c603cd47f7)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 870cf57635 Fix flaky gauge value issue (#13679) add c603cd47f7 Fix query response time metric (#13661) No new revisions were added by this update. Summary of changes: .../pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java | 7 +++ 1 file changed, 7 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (a852c8a424 -> a5c728f549)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from a852c8a424 Update ORC and Hive dependency versions in the license binary file (#12986) add a5c728f549 Add back profile for shade (#12979) No new revisions were added by this update. Summary of changes: pinot-clients/pinot-jdbc-client/pom.xml | 15 ++- pinot-common/pom.xml | 11 ++- pinot-core/pom.xml | 11 +++ pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 16 +++- .../pinot-stream-ingestion/pinot-kinesis/pom.xml | 16 +++- pinot-spi/pom.xml| 11 +++ 6 files changed, 76 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
svn commit: r68065 - /dev/pinot/apache-pinot-1.1.0-rc1/ /release/pinot/apache-pinot-1.1.0/
Author: mcvsubbu Date: Fri Mar 22 22:26:40 2024 New Revision: 68065 Log: Step 1 to finalize 1.1.0-rc1 Moving apache-pinot-1.1.0-rc1 from dev to release Added: release/pinot/apache-pinot-1.1.0/ - copied from r68064, dev/pinot/apache-pinot-1.1.0-rc1/ Removed: dev/pinot/apache-pinot-1.1.0-rc1/ - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
svn commit: r66884 - /release/pinot/KEYS
Author: mcvsubbu Date: Sat Jan 27 01:33:58 2024 New Revision: 66884 Log: Adding vivek iyer as committer Modified: release/pinot/KEYS Modified: release/pinot/KEYS == --- release/pinot/KEYS (original) +++ release/pinot/KEYS Sat Jan 27 01:33:58 2024 @@ -751,3 +751,62 @@ Ff4FErS11Jqkq6IdbgHqjHGsMpYXDXPsnTtgHwSr 3Qg5Y6s= =EDvf -END PGP PUBLIC KEY BLOCK- +pub rsa4096 2024-01-17 [SC] + 362A8E75B68E452C7DC8C8601D2B6F73AEA25682 +uid [ultimate] Vivek Iyer Vaidyanathan Iyer +sig 31D2B6F73AEA25682 2024-01-17 [self-signature] +sub rsa4096 2024-01-17 [E] +sig 1D2B6F73AEA25682 2024-01-17 [self-signature] + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBGWoGZ4BEAD4rLYHOyOgxqaCdx3fqyAkUaK8C8fJBZXaQIstjMRuHX10+bbW +3Si3rYFlz5IlcvBO5kyiyQJHL3HMxKZTNk71cE02qnbkKgsn6ym4SZ+WflV0Nxvu +jYl5LfQWzOVIFfa02yJrMuP+b4AGkbZjaVU2s335uhNjEE863lnAxmb7ZIgOw34G +jzVcDz/Hls2/UwAOK1wndCa/PFNg0GkraOr1G9hPwuI+bgP2wp3QGPzHQVJ79N/F +ubUn3yGNvRVIwHbQ15r/QFyzpMNp6LO1k9RfGjAwu/ODSRu29v0cxkcCVHi0XF9r +jlIJq3H4I/opCtEagMErsAjL5xXwM1hznjTXFMwLDvuAOo4PSphqVk+Wg83cEkmL +3AiR3yULZwI57g330QFYIRl/O/TOw9ERYqEwo6bqY5XFy9JqUIg8EPXjRujjGk/P +jqWtW4C12toVQf3NfCyamkMIun8/2YCiLFrOIIZR4YDJ1+Y9JaUgVn8xEO0CDAZH +Ec+WsVmYU+upUlmHBxbDLWNhTICUJCF5pMIA8rflClj/7evNagzYrxtA8XiqNSlc +k5DbivQ0IMMpoSuccHK+4PjhkgfC64B5xZVH9tK0VuzlkFhJEzzwuitFEzk4KExS +pdjMFhUPYBWtsqfTz4ISg0saZkAYkGH+SB74odiuDoakQXKEjTJ9Cb/W+wARAQAB +tDRWaXZlayBJeWVyIFZhaWR5YW5hdGhhbiBJeWVyIDx2dml2ZWtpeWVyQGFwYWNo +ZS5vcmc+iQJRBBMBCAA7FiEENiqOdbaORSx9yMhgHStvc66iVoIFAmWoGZ4CGwMF +CwkIBwICIgIGFQoJCAsCBBYCAwECHgcCF4AACgkQHStvc66iVoJA9xAA5GSbCUvf +7zzFKr2AQqKpjyLMIvhOGDqSj3MInLfO4PTl5IxuUD0FfOomJXpgmzEoOSqXgc0e +bqqCLOO/Ev0JbJR6PFicm524K+rT7qUeW+TnUPAoefEVCDajf3jMTipKNNagu6YS +L/JLI9B5VuGkOHhZD3rE6zpcSc5wrXC2uvQCEB2AN/8u2vHpwsDPXrWj/gJpeQxg +qOy91RhKGhxE7CQpGfWp/cntnBAmo5NA87t0EMsWaUJNsO/JpaKxlTdyup3lsLoC +yQSWQcmNhhvxgoOuZjbM0HM322myII0BPwk5dtsUcCwvslv9k6nqnybUk+T8d/ma +MUH/grmOHJqLz4349LEnlgkJNkN/Nfm06sH9QobbtUUayRPc2n3tAzMjkbWGBecY +LFxVO38JETFhi9f1Zn6pULQ00pfUYX+HAXDNwd0cnmCIm3BDYxy8xRuEls9xyQLf +zYl1a0r/gBA+USi0zblGmOGdC593T5e1ol9CdhK1RQ0S4WoMLwYuijSTncFUHayH +z+AAVOTbYH1MGDs2LjeHWt+PENfvy9cb586jwt9LG5F05cY9OLpiUAqo+ndxp02T +01daKfOtpf0t0ksSjIrWNg/i+3rTkknVEcorDFRGQckG2PcuSGK1tPs24ot9QxZu +0rm3Pmk1W9cwX+IHRB3scONYc9eJH4eQX/S5Ag0EZagZngEQALIA5huBnlsAJrnc +q9/5wmXCREB3651WAkBOB+ajH03VAkRfOsEsrh9KQG0Kez+zyurKONit7W4bjt4o +slGrDugFETmpe3q8EVTPzE99AC2TncyA36ADCwFHhru57jAUhVm36ucd8GdXutxH +s3f97m8trc847oa6IRYZ81xUqC0Y8XviuWWWtHCsk1mZ6XJ/zWOv1fsoIySf5izl +UPqrzXbj1FfAKNmXwx1YMVB4aNFHDF5usnAbOjKfpf2oo7sSNdblE6SXeSUpjed2 +8RqJ5fFEFQItzn0E+UHuEHJTkh7ccBre/wWlTexRFu4UyUVftlkWaYREwTa/3SCN +O8r/vF04OzlM+iT1uZ4MVofa4NPaDPdno0+lbdwBP5J0hq1XlLdHmi961nbjPC3d +o9Zvi+PDDW4RDMA8P/Iv5t8fd4sTLjkPt6rRAVcP3F27fydTROpuMgCeJvPbahFw +C47L0vzFzRPOMUB76eofUbaIQNdIphORMZXRyr6aCQX1kUVSNUOD8p4NHYJg6MDQ +5WHw15ajjUTUf+eo2C/Q1Uj+kVOn712KF/R/NiXDUlcq/zzC1Z74Sle0ApivsX2X +6zxEX5Uls1WTz60fQCZzqIirfyQPCM5m6N20XlpqBE2egjQyM9+paLk/XejBbUhV +6w2ks8D3/5pZXSS30pBa5aYgnmOHABEBAAGJAjYEGAEIACAWIQQ2Ko51to5FLH3I +yGAdK29zrqJWggUCZagZngIbDAAKCRAdK29zrqJWgj5xD/0cHjBgNFTie8G57wnD +8CHEIrUyrFi+lmjdhsp0So2gEwqJl29wN68xfgUYflPLYlcZ1qRBH5Q4uL/ICEEP +npls0okZhM2fN32qLo4vxYK7xVFAEwbMNYOyTbzWQ29PkJvos3wW5xvadmfFzw/P +7e20VTELQezk0HyQYc0/FqMHu7eS6YHkIFFlJ1XwSCipNYYDAA08VYiF6tE20E0H +oeip5D/FsDp7jDz2O3SWf8cktLM5xE3qSa7xnz8Q3cP0HSKJOJAibRU2/n/x1AEV +UJo9ouTChHG66A0m3YrrfDfowv4xvQy7yIMlWCmyKVcDc43wIEZQ1UbuWK9Llz3H +GwWl/JGB/3RCcoaHfTXPp7cYU712N61AJ5adasCqn5hDAIcTIHa75AA7btgF1oBj +NFZMYRmnoK156lYZ+0npdMv5EI4T9SHEmxNCsfKXbgP5nVyd1g7SBQeiZiywlPsq +goUcLV0uRx2VnK33+B9i1dGjWWq2LlYOt1MhrdT6rBBhrNkUPcDV2Bjlib0c6Zzq +QVDONbG9yxSDHFnQxEkodlePqP4gm8KBnVPa2fKC3JsVks1w4ncag+4796o8W8Zg +GprftCerfzoNS+vmRzTA+9pSyW3hTb6swvf4qsfz+4ohb220gM84Zvh+QSphpZZ8 +EMX6EpYToJFmb2/O6suRoCBCeQ== +=WdIu +-END PGP PUBLIC KEY BLOCK- - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Fix incorrect handling of consumer creation errors (#12045)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 6fe2d25c21 Fix incorrect handling of consumer creation errors (#12045)
6fe2d25c21 is described below
commit 6fe2d25c21fdc9b8364b6e594d96b78e907717e6
Author: Subbu Subramaniam
AuthorDate: Tue Nov 28 13:05:13 2023 -0800
Fix incorrect handling of consumer creation errors (#12045)
* Fix incorrect handling of consumer creation errors
The current handling of exceptions during creation of a consumer is
incorrect, since the ExternalView for the segment remains in ERROR
state while we specify the IdealState to be OFFLINE. This happens
for one replica, while other replicas may consume fine and reach
ONLINE state eventually. At that time, however, the particular
segment that had problem consuming is not able to transition to
ONLINE since a transition from ERROR to ONLINE is not suppored by Helix.
A partition state of ERROR is a special state in Helix. Helix does not
work the same way moving from ERROR to other states. Instead,
Helix provides an admin API to reset the state of a partition from ERROR
to its starting state (which in our case is OFFLINE). When this reset
API is invoked, a state transition message of ERROR to StartingState
is sent to the specific instance that hosts the partition in question.
If the participant's currentstate is not ERROR, then this message
is discarded automatically (and Pinot will never see it). Otherwise,
it is passed on to Pinot and we have a transition from ERROR to OFFLINE.
Tested by manually inserting an exception in the consumer creation
code ad observing that externalview changes to ERROR, and then later
onto OFFLINE.
* Ignore Helix expection on reset call
Helix throws an exceptioin if we call reset on a partition that is not in
ERROR
state in ExternalView. So, we need to ignore any exception that helix sends
out
on the reset call (it is a best-effort call anyway).
* Address linter check
* Added tests
* Addressed review comments
* Addressed review comment to fix space
---
.../controller/helix/ControllerRequestClient.java | 10 ++
.../realtime/PinotLLCRealtimeSegmentManager.java | 16 +++
.../realtime/RealtimeSegmentDataManager.java | 55 +++-
.../tests/BaseRealtimeClusterIntegrationTest.java | 6 +
.../tests/LLCRealtimeClusterIntegrationTest.java | 149 +
.../utils/builder/ControllerRequestURLBuilder.java | 4 +
6 files changed, 233 insertions(+), 7 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/ControllerRequestClient.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/ControllerRequestClient.java
index ff27954f70..e37172e3b3 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/ControllerRequestClient.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/ControllerRequestClient.java
@@ -325,6 +325,16 @@ public class ControllerRequestClient {
}
}
+ public void runPeriodicTask(String taskName)
+ throws IOException {
+try {
+ HttpClient.wrapAndThrowHttpException(_httpClient.sendGetRequest(new URL(
+ _controllerRequestURLBuilder.forPeriodTaskRun(taskName)).toURI()));
+} catch (HttpErrorStatusException | URISyntaxException e) {
+ throw new IOException(e);
+}
+ }
+
protected String getBrokerTenantRequestPayload(String tenantName, int
numBrokers) {
return new Tenant(TenantRole.BROKER, tenantName, numBrokers, 0,
0).toJsonString();
}
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
index 90cd05d16b..092e47cced 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
@@ -821,6 +821,22 @@ public class PinotLLCRealtimeSegmentManager {
_controllerMetrics.addMeteredTableValue(realtimeTableName,
ControllerMeter.LLC_ZOOKEEPER_UPDATE_FAILURES, 1L);
throw e;
}
+// We know that we have successfully set the idealstate to be OFFLINE.
+// We can now do a best effort to reset the externalview to be OFFLINE if
it is in ERROR state.
+// If the externalview is not in error state, then this reset will be
ignored by the helix participant
+// in the server when it receives the ERROR to OFFLINE state transition.
+// Helix throws an exception
[pinot] branch master updated: Make PinotTableRestletResourceTest java8 compatible (#11287)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new bc4a225428 Make PinotTableRestletResourceTest java8 compatible (#11287)
bc4a225428 is described below
commit bc4a2254286338c4051bcd046a22c425bc908fbc
Author: Dino Occhialini
AuthorDate: Mon Aug 7 15:02:07 2023 -0700
Make PinotTableRestletResourceTest java8 compatible (#11287)
This commit fixes java8 build compatibility that is broken in #11017
Specifically, `Map.of` is only available in java 9+.
In this case, this code:
```java
TableConfigBuilder(TableType.REALTIME)
.setStreamConfigs(Map.of("stream.type", "foo", "consumer.type",
"lowlevel"));
```
Is actually not used since we are later calling the following in setup:
```java
StreamConfig streamConfig =
FakeStreamConfigUtils.getDefaultLowLevelStreamConfigs();
_realtimeBuilder...setStreamConfigs(streamConfig.getStreamConfigsMap());
```
---
.../org/apache/pinot/controller/api/PinotTableRestletResourceTest.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotTableRestletResourceTest.java
b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotTableRestletResourceTest.java
index f0d17de26b..7203da3819 100644
---
a/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotTableRestletResourceTest.java
+++
b/pinot-controller/src/test/java/org/apache/pinot/controller/api/PinotTableRestletResourceTest.java
@@ -58,8 +58,7 @@ public class PinotTableRestletResourceTest extends
ControllerTest {
private static final String OFFLINE_TABLE_NAME = "testOfflineTable";
private static final String REALTIME_TABLE_NAME = "testRealtimeTable";
private final TableConfigBuilder _offlineBuilder = new
TableConfigBuilder(TableType.OFFLINE);
- private final TableConfigBuilder _realtimeBuilder = new
TableConfigBuilder(TableType.REALTIME)
- .setStreamConfigs(Map.of("stream.type", "foo", "consumer.type",
"lowlevel"));
+ private final TableConfigBuilder _realtimeBuilder = new
TableConfigBuilder(TableType.REALTIME);
private String _createTableUrl;
@BeforeClass
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated: Add minion observability for segment upload/download failures (#10978)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 2d30e28bed Add minion observability for segment upload/download
failures (#10978)
2d30e28bed is described below
commit 2d30e28bedea29d264b0d7946ebb3f0e2cf1b5cc
Author: Subbu Subramaniam
AuthorDate: Mon Jun 26 19:25:26 2023 -0700
Add minion observability for segment upload/download failures (#10978)
* Add minion observability for segment upload/download failures
Currently, minions do not provide observability into upload or download
failures. Added mechanism to log errors and bump metrics when either
upload or download fails, so that operators can set alerts on these
metrics to detect the inconsistent state quickly and remediate if possible.
Issue #10973
* Style fix
---
.../apache/pinot/common/metrics/MinionMeter.java | 4 ++-
.../tasks/BaseSingleSegmentConversionExecutor.java | 31 +++---
2 files changed, 30 insertions(+), 5 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionMeter.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionMeter.java
index 101a950389..376f86e55e 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionMeter.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionMeter.java
@@ -29,7 +29,9 @@ public enum MinionMeter implements AbstractMetrics.Meter {
NUMBER_TASKS_COMPLETED("tasks", false),
NUMBER_TASKS_CANCELLED("tasks", false),
NUMBER_TASKS_FAILED("tasks", false),
- NUMBER_TASKS_FATAL_FAILED("tasks", false);
+ NUMBER_TASKS_FATAL_FAILED("tasks", false),
+ SEGMENT_UPLOAD_FAIL_COUNT("segments", false),
+ SEGMENT_DOWNLOAD_FAIL_COUNT("segments", false);
private final String _meterName;
private final String _unit;
diff --git
a/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
b/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
index bc440e0c5e..51c7f98543 100644
---
a/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
+++
b/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/BaseSingleSegmentConversionExecutor.java
@@ -32,13 +32,17 @@ import org.apache.http.HttpHeaders;
import org.apache.http.NameValuePair;
import org.apache.http.message.BasicHeader;
import org.apache.http.message.BasicNameValuePair;
+import org.apache.pinot.common.Utils;
import org.apache.pinot.common.auth.AuthProviderUtils;
import
org.apache.pinot.common.metadata.segment.SegmentZKMetadataCustomMapModifier;
+import org.apache.pinot.common.metrics.MinionMeter;
+import org.apache.pinot.common.metrics.MinionMetrics;
import org.apache.pinot.common.utils.FileUploadDownloadClient;
import org.apache.pinot.common.utils.TarGzCompressionUtils;
import org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory;
import org.apache.pinot.core.common.MinionConstants;
import org.apache.pinot.core.minion.PinotTaskConfig;
+import org.apache.pinot.minion.MinionContext;
import org.apache.pinot.minion.event.MinionEventObserver;
import org.apache.pinot.minion.event.MinionEventObservers;
import org.apache.pinot.minion.exception.TaskCancelledException;
@@ -60,6 +64,7 @@ public abstract class BaseSingleSegmentConversionExecutor
extends BaseTaskExecut
// Tracking finer grained progress status.
protected PinotTaskConfig _pinotTaskConfig;
protected MinionEventObserver _eventObserver;
+ protected final MinionMetrics _minionMetrics =
MinionContext.getInstance().getMinionMetrics();
/**
* Converts the segment based on the given task config and returns the
conversion result.
@@ -101,7 +106,14 @@ public abstract class BaseSingleSegmentConversionExecutor
extends BaseTaskExecut
_eventObserver.notifyProgress(_pinotTaskConfig, "Downloading segment
from: " + downloadURL);
File tarredSegmentFile = new File(tempDataDir, "tarredSegment");
LOGGER.info("Downloading segment from {} to {}", downloadURL,
tarredSegmentFile.getAbsolutePath());
- SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(downloadURL,
tarredSegmentFile, crypterName);
+ try {
+SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(downloadURL,
tarredSegmentFile, crypterName);
+ } catch (Exception e) {
+_minionMetrics.addMeteredTableValue(tableNameWithType,
MinionMeter.SEGMENT_DOWNLOAD_FAIL_COUNT, 1
[pinot] branch master updated: Change list.of to immutableList.of to solve java 8 compilation (#10929)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 85bb1fab54 Change list.of to immutableList.of to solve java 8
compilation (#10929)
85bb1fab54 is described below
commit 85bb1fab54d44e847195e35e3e0b117022985d36
Author: Vivek Iyer Vaidyanathan
AuthorDate: Thu Jun 15 21:13:39 2023 -0700
Change list.of to immutableList.of to solve java 8 compilation (#10929)
---
.../segment/processing/framework/SegmentProcessorFrameworkTest.java| 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/framework/SegmentProcessorFrameworkTest.java
b/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/framework/SegmentProcessorFrameworkTest.java
index e977ad364c..c0deaf85b5 100644
---
a/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/framework/SegmentProcessorFrameworkTest.java
+++
b/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/framework/SegmentProcessorFrameworkTest.java
@@ -18,6 +18,7 @@
*/
package org.apache.pinot.core.segment.processing.framework;
+import com.google.common.collect.ImmutableList;
import java.io.File;
import java.io.IOException;
import java.net.URL;
@@ -204,7 +205,7 @@ public class SegmentProcessorFrameworkTest {
SegmentProcessorConfig config =
new
SegmentProcessorConfig.Builder().setTableConfig(tableConfig).setSchema(schema).build();
-SegmentProcessorFramework framework = new
SegmentProcessorFramework(config, workingDir, List.of(reader));
+SegmentProcessorFramework framework = new
SegmentProcessorFramework(config, workingDir, ImmutableList.of(reader));
List outputSegments = framework.process();
assertEquals(outputSegments.size(), 1);
ImmutableSegment segment =
ImmutableSegmentLoader.load(outputSegments.get(0), ReadMode.mmap);
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated (434d9733c3 -> 2e6f1e63f0)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 434d9733c3 Add a comment to and a unit test for the group by expressions deduplication logic (#10803) add 2e6f1e63f0 Use min of (retry count, number of controller addresses) as retry count when fetch segment (#10804) No new revisions were added by this update. Summary of changes: .../pinot/common/utils/fetcher/HttpSegmentFetcher.java | 12 +--- 1 file changed, 9 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated (27eb9601fa -> 579082c803)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 27eb9601fa [multisage] fix set op issue (#10764) add 579082c803 Compress idealstate according to estimated size (#10766) No new revisions were added by this update. Summary of changes: .../pinot/common/utils/helix/HelixHelper.java | 36 +- .../pinot/controller/BaseControllerStarter.java| 1 + .../apache/pinot/controller/ControllerConf.java| 10 ++ 3 files changed, 46 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated: [feature] Add a Tracker class to support aggregate-worst case consumption delay… (#9994)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 6a216c5a4f [feature] Add a Tracker class to support aggregate-worst case consumption delay… (#9994) 6a216c5a4f is described below commit 6a216c5a4fa8cc91dcdbb4c40949d2d2b83e6453 Author: Juan Gomez AuthorDate: Mon Jan 9 17:43:28 2023 -0800 [feature] Add a Tracker class to support aggregate-worst case consumption delay… (#9994) * Add a Tracker class to support aggregate-worst case consumption delay in Pinot Early draft for review, no significant unitest or end to end testing done yet. Missing: code to read ideal state, unitets. -Changed code to not remove but verify a partition when CONSUMING to OFFLINE transition comes Tests: make sure server, core tests pass. -Added support for reading ideal state -Added some unitests for tracker object -Added more unitests that deal with partition removal. -Removed caching the maximum to simplefy the code. -Use Concurrent hashes instead of synchronized methods. -Renamed some variables for consistency -Refactored timeout code -Fix failing unitests and a number of renames -Use int for partitionGroupIds instead of longs -Some renames and minor edits. -Move some of uses of partition id to be integers for consistency with previous code. Remove use of type cast by adding interfaces to super class. * Remove initial delay of timer thread from configurable parameters, clean throw message and update comments to reflect new method names. * Added support for per partition metrics and some renames. Added support for prefix to metric name so we can add multiple trackers and metrics per table. * Add code to support disabling emitting per partition or aggregate metric. * Do per-batch sampling of delay metric and not per event for efficiency reasons. Some rename to avoid the Pinot word in name of ingestion delay. * Add Navina's code to not sample partition delay during catching up phase. * Rename Millis to Ms and rearrange final private to private final * Rename Consumption Delay to ingestion delay, remove unnecessary casts, factor out zeroing out the ingestion delay. * Use anonymous class for timer tasks, some final variables and add TODO for some issues we noted in the code while working on this. * Remove checks for _ingestionDelayTracker being null as a number of other things break before if that happens. Rollback method to set delay to zero after oflline discussion. * Consolidate segement state transition actions in a single method, some minor edits to comments to reflect we now track per partition ingestion delay * Roll back change where we use one interface for all segement state transitions, code is simpler with interface per transition, also helps with consistency with other parts of the code. * Optimization to only check ideal state when we really need to. * Cleanup some comments with stale references, some minor refactors. * Consolidate all ingestion delay measures under the same Gauge. * Eliminte the use of optional aging and fix tests to use fixed clocks and get predictable results. * Remove the per-partition metric mute and use server ready to serve queries as signal to start collecting metrics. * Address most of the last round of comments. * Remove isServerReady code from Segment data provider to keep code simpler, encapsulate in IngestionDelay class instead. * Use CONSUMING segements to determine hosted partitions instead of ONLINE, getSegmentsInGivenStateForThisInstance returns set instead of list. * Added few unitests for per partition measures, few comments edits * Removing aggregate metric and some other things requested to simplify the change. --- .../pinot/common/metrics/AbstractMetrics.java | 31 +++ .../apache/pinot/common/metrics/ServerGauge.java | 4 +- .../core/data/manager/InstanceDataManager.java | 8 + .../manager/offline/TableDataManagerProvider.java | 10 +- .../manager/realtime/IngestionDelayTracker.java| 308 + .../realtime/LLRealtimeSegmentDataManager.java | 19 +- .../manager/realtime/RealtimeTableDataManager.java | 68 - .../realtime/IngestionDelayTrackerTest.java| 204 ++ .../local/data/manager/TableDataManager.java | 16 ++ .../local/utils/tablestate/TableStateUtils.java| 45 ++- .../server/starter/helix/BaseServerStarter.java| 4 +- .../starter/helix/HelixInstanceDataManager.java| 9 +- .../SegmentOnlineOfflineStateModelFactory.java | 9 +- 13 files changed, 715 insertions(+
[pinot] branch master updated (fad3de8023 -> fd5c942fc8)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
from fad3de8023 Do not use aggregation result for distinct query in
IntermediateResultsBlock (#9262)
add fd5c942fc8 Add metrics to track controller segment download and upload
requests in progress (#9258)
No new revisions were added by this update.
Summary of changes:
.../pinot/common/metrics/ControllerGauge.java | 9 +-
.../InflightRequestMetricsInterceptor.java | 85 +++
.../resources/LLCSegmentCompletionHandlers.java| 3 +
.../PinotSegmentUploadDownloadRestletResource.java | 11 +++
.../api/resources/TrackInflightRequestMetrics.java | 18 ++--
.../TrackedByGauge.java} | 15 ++--
.../InflightRequestMetricsInterceptorTest.java | 99 ++
7 files changed, 221 insertions(+), 19 deletions(-)
create mode 100644
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/InflightRequestMetricsInterceptor.java
copy
pinot-spi/src/main/java/org/apache/pinot/spi/annotations/metrics/MetricsFactory.java
=>
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/TrackInflightRequestMetrics.java
(71%)
copy
pinot-controller/src/main/java/org/apache/pinot/controller/api/{access/Authenticate.java
=> resources/TrackedByGauge.java} (68%)
create mode 100644
pinot-controller/src/test/java/org/apache/pinot/controller/api/resources/InflightRequestMetricsInterceptorTest.java
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated: Handle unknown magic byte error in Confluent Avro decoder (#9045) (#9051)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new bbec0a8070 Handle unknown magic byte error in Confluent Avro decoder
(#9045) (#9051)
bbec0a8070 is described below
commit bbec0a8070ce1d689a382195b291d68ca945c040
Author: Daniel del Castillo
AuthorDate: Sun Jul 24 16:49:21 2022 +0100
Handle unknown magic byte error in Confluent Avro decoder (#9045) (#9051)
* Handle unknown magic byte error in Confluent Avro decoder (#9045)
* Update StreamMessageDecoder documentation
---
pinot-integration-tests/pom.xml| 25 ++
...ssageDecoderRealtimeClusterIntegrationTest.java | 313 +
.../schemaregistry/SchemaRegistryStarter.java | 105 +++
...aConfluentSchemaRegistryAvroMessageDecoder.java | 33 ++-
.../pinot/spi/stream/StreamMessageDecoder.java | 9 +-
5 files changed, 481 insertions(+), 4 deletions(-)
diff --git a/pinot-integration-tests/pom.xml b/pinot-integration-tests/pom.xml
index a67a222d5b..3da652cff0 100644
--- a/pinot-integration-tests/pom.xml
+++ b/pinot-integration-tests/pom.xml
@@ -37,6 +37,7 @@
0.2.19
3.0.0
2.14.28
+1.17.3
@@ -323,6 +324,30 @@
pinot-yammer
${project.version}
+
+ org.apache.pinot
+ pinot-confluent-avro
+ ${project.version}
+ test
+
+
+ org.testcontainers
+ testcontainers
+ ${testcontainers.version}
+ test
+
+
+ net.java.dev.jna
+ jna
+
+
+
+
+ org.testcontainers
+ kafka
+ ${testcontainers.version}
+ test
+
cloud.localstack
localstack-utils
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/KafkaConfluentSchemaRegistryAvroMessageDecoderRealtimeClusterIntegrationTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/KafkaConfluentSchemaRegistryAvroMessageDecoderRealtimeClusterIntegrationTest.java
new file mode 100644
index 00..21627741f1
--- /dev/null
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/KafkaConfluentSchemaRegistryAvroMessageDecoderRealtimeClusterIntegrationTest.java
@@ -0,0 +1,313 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.integration.tests;
+
+import com.google.common.primitives.Longs;
+import io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig;
+import java.io.File;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import org.apache.avro.file.DataFileStream;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.commons.io.FileUtils;
+import org.apache.http.HttpStatus;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.FileUploadDownloadClient;
+import org.apache.pinot.controller.ControllerConf;
+import
org.apache.pinot.integration.tests.kafka.schemaregistry.SchemaRegistryStarter;
+import org.apache.pinot.plugin.inputformat.avro.AvroUtils;
+import
org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.env.PinotConfiguration;
+import org.apache.pinot.spi.stream.StreamConfigProperties;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.uti
[pinot] branch master updated: Adjust comparison of numDocsScanned (#8433)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new deaccf7 Adjust comparison of numDocsScanned (#8433)
deaccf7 is described below
commit deaccf776845d7dc670f686b6d12bec5bf27f266
Author: Subbu Subramaniam
AuthorDate: Wed Mar 30 10:03:46 2022 -0700
Adjust comparison of numDocsScanned (#8433)
* Adjust comparison of numDocsScanned
Number of documents scanned can be lower than expected when comparing
results of compatibility tests. We don't like it to be higher since that
may indicate a performance regression.
Changed the code to not expect numDocsScanned to be equal all the time
* Fix style errors
* Fixed log statements
---
.../pinot/common/utils/SqlResultComparator.java| 66 --
1 file changed, 36 insertions(+), 30 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/SqlResultComparator.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/SqlResultComparator.java
index d6357a7..c9f2481 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/SqlResultComparator.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/SqlResultComparator.java
@@ -126,29 +126,35 @@ public class SqlResultComparator {
*/
if (expected.has(FIELD_IS_SUPERSET) &&
expected.get(FIELD_IS_SUPERSET).asBoolean(false)) {
return areElementsSubset(actualElementsSerialized,
expectedElementsSerialized);
-} else {
- if (!areLengthsEqual(actual, expected)) {
-return false;
- }
- /*
- * Pinot server do some early termination optimization (process in
parallel and early return if get enough
- * documents to fulfill the LIMIT and OFFSET requirement) for queries:
- * - selection without order by
- * - selection with order by (by sorting the segments on min-max value)
- * - DISTINCT queries.
- * numDocsScanned is non-deterministic for those queries so
numDocsScanned comparison should be skipped.
- *
- * NOTE: DISTINCT queries are modeled as non-selection queries during
query processing, but all DISTINCT
- * queries are selection queries during Calcite parsing (DISTINCT
queries are selection queries with
- * selectNode.getModifierNode(SqlSelectKeyword.DISTINCT) != null).
- */
- if (!isSelectionQuery(query) && !areNumDocsScannedEqual(actual,
expected)) {
+}
+if (!areLengthsEqual(actual, expected)) {
+ return false;
+}
+boolean areResultsEqual = isOrderByQuery(query) ?
areOrderByQueryElementsEqual(actualRows, expectedRows,
+actualElementsSerialized, expectedElementsSerialized, query)
+: areNonOrderByQueryElementsEqual(actualElementsSerialized,
expectedElementsSerialized);
+/*
+ * Pinot servers implement early termination optimization (process in
parallel and early return if we get enough
+ * documents to fulfill the LIMIT and OFFSET requirement) for queries for
the following cases:
+ * - selection without order by
+ * - selection with order by (by sorting the segments on min-max value)
+ * - DISTINCT queries.
+ * numDocsScanned is non-deterministic for those queries so numDocsScanned
comparison should be skipped.
+ *
+ * In other cases, we accept if numDocsScanned is better than the expected
one, since that is indicative of
+ * performance improvement.
+ *
+ * NOTE: DISTINCT queries are modeled as non-selection queries during
query processing, but all DISTINCT
+ * queries are selection queries during Calcite parsing (DISTINCT queries
are selection queries with
+ * selectNode.getModifierNode(SqlSelectKeyword.DISTINCT) != null).
+ */
+if (areResultsEqual) {
+ // Results are good, check for any metadata differences here.
+ if (!isSelectionQuery(query) && !isNumDocsScannedBetter(actual,
expected)) {
return false;
}
- return isOrderByQuery(query) ? areOrderByQueryElementsEqual(actualRows,
expectedRows, actualElementsSerialized,
- expectedElementsSerialized, query)
- : areNonOrderByQueryElementsEqual(actualElementsSerialized,
expectedElementsSerialized);
}
+return areResultsEqual;
}
private static boolean areOrderByQueryElementsEqual(ArrayNode
actualElements, ArrayNode expectedElements,
@@ -275,7 +281,7 @@ public class SqlResultComparator {
return true;
}
- public static boolean hasExceptions(JsonNode actual) {
+ private static boolean hasExceptions(JsonNode actual) {
if (!actual.get(FIELD_EXCEPTIONS).isEmpty()) {
LOGGER.error("Got exception: {} when querying!",
actual.get(FIELD_EXCEPTIONS));
return true;
@@ -369,34 +375,34 @@ public class
[pinot] branch master updated: Disable recursion in PinotFS copy (#8162)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 1382d29 Disable recursion in PinotFS copy (#8162)
1382d29 is described below
commit 1382d293236d40e80d1652729fdb1632f92f7e48
Author: Subbu Subramaniam
AuthorDate: Tue Feb 8 15:10:30 2022 -0800
Disable recursion in PinotFS copy (#8162)
* Disable recursion in PinotFS copy
All uses of PinotFS copy API involve copying a tarred segment and
untarring it. So, copying a directory recursively will not work (the
untar will fail). It also results in wastage of effort in copying
across file systems.
Also disabled the file scheme in during segment upload on the controller,
since the URL based upload is meant to provide an external URL to be
picked up by the controller.
* Removed redundant comment
* Fix to add a different API for recursive copy
* Fix lint errors
---
.../utils/fetcher/SegmentFetcherFactory.java | 1 +
.../PinotSegmentUploadDownloadRestletResource.java | 5
.../hadoop/HadoopSegmentGenerationJobRunner.java | 6 ++---
.../pinot/plugin/filesystem/HadoopPinotFS.java | 15 +++
.../apache/pinot/spi/filesystem/LocalPinotFS.java | 25 ++-
.../org/apache/pinot/spi/filesystem/PinotFS.java | 16 ++--
.../pinot/spi/filesystem/LocalPinotFSTest.java | 29 --
7 files changed, 57 insertions(+), 40 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactory.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactory.java
index c6d71e8..743cd1b 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactory.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/SegmentFetcherFactory.java
@@ -144,6 +144,7 @@ public class SegmentFetcherFactory {
private void fetchSegmentToLocalInternal(URI uri, File dest)
throws Exception {
+// caller untars
getSegmentFetcher(uri.getScheme()).fetchSegmentToLocal(uri, dest);
}
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
index 484bc30..253a3fb 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
@@ -381,6 +381,11 @@ public class PinotSegmentUploadDownloadRestletResource {
}
LOGGER.info("Downloading segment from {} to {} for table {}",
currentSegmentLocationURI, destFile.getAbsolutePath(),
tableName);
+URI uri = new URI(currentSegmentLocationURI);
+if (uri.getScheme().equalsIgnoreCase("file")) {
+ throw new ControllerApplicationException(LOGGER, "Unsupported URI: " +
currentSegmentLocationURI,
+ Response.Status.BAD_REQUEST);
+}
SegmentFetcherFactory.fetchSegmentToLocal(currentSegmentLocationURI,
destFile);
}
diff --git
a/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
b/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
index 4cbfdf5..0608fd7 100644
---
a/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
+++
b/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
@@ -259,7 +259,7 @@ public class HadoopSegmentGenerationJobRunner extends
Configured implements Inge
job.getConfiguration().setBoolean(MRJobConfig.MAP_SPECULATIVE, false);
// But we have to copy ourselves to HDFS, and add us to the distributed
cache, so
- // that the mapper code is available.
+ // that the mapper code is available.
addMapperJarToDistributedCache(job, outputDirFS, stagingDirURI);
org.apache.hadoop.conf.Configuration jobConf = job.getConfiguration();
@@ -277,7 +277,7 @@ public class HadoopSegmentGenerationJobRunner extends
Configured implements Inge
// In order to ensure pinot plugins would be loaded to each worker, this
method
// tars entire plugins directory and set this file into Distributed
cache.
// Then each mapper job
[pinot] branch master updated: Fixed segment upload error return (#7957)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 3aa4f5d Fixed segment upload error return (#7957)
3aa4f5d is described below
commit 3aa4f5dab085f1d2f55ea37a6790477e97ddb27e
Author: Subbu Subramaniam
AuthorDate: Fri Dec 24 13:50:16 2021 -0800
Fixed segment upload error return (#7957)
* Fixed segment upload error return
* Addressed review comment
---
.../api/resources/PinotSegmentUploadDownloadRestletResource.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
index d6d3889..33a1d55 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
@@ -308,7 +308,7 @@ public class PinotSegmentUploadDownloadRestletResource {
throw e;
} catch (Exception e) {
_controllerMetrics.addMeteredGlobalValue(ControllerMeter.CONTROLLER_SEGMENT_UPLOAD_ERROR,
1L);
- throw new ControllerApplicationException(LOGGER, "Caught internal server
exception while uploading segment",
+ throw new ControllerApplicationException(LOGGER, "Exception while
uploading segment: " + e.getMessage(),
Response.Status.INTERNAL_SERVER_ERROR, e);
} finally {
FileUtils.deleteQuietly(tempEncryptedFile);
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated (6037cac -> f27cf0b)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git. from 6037cac Disable JsonStatementOptimizer. (#7919) add f27cf0b Fix performance problem of base chunk forward index write (#7930) No new revisions were added by this update. Summary of changes: .../writer/impl/BaseChunkSVForwardIndexWriter.java | 56 ++ 1 file changed, 24 insertions(+), 32 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated (c8071f7 -> aa2da07)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git. from c8071f7 App assembler script should check that the correct delimiter is used for plugin related env vars (#7905) add aa2da07 BUG FIX: semaphore issue in consuming segments (#7886) No new revisions were added by this update. Summary of changes: .../realtime/LLRealtimeSegmentDataManager.java | 134 - .../realtime/LLRealtimeSegmentDataManagerTest.java | 2 +- 2 files changed, 77 insertions(+), 59 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated (9ba03e4 -> dfb7fee)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git. from 9ba03e4 Fix thread safety issue and add cache to EmptySegmentPruner (#7828) add dfb7fee Make split commit the default commit protocol (#7780) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/pinot/controller/ControllerConf.java | 2 +- .../pinot/server/starter/helix/HelixInstanceDataManagerConfig.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated: Add config for enabling realtime offset based consumption status checker (#7753)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 6611ff8 Add config for enabling realtime offset based consumption
status checker (#7753)
6611ff8 is described below
commit 6611ff82f46ea837e6bd8edefd46ccc16d4e2002
Author: Sajjad Moradi
AuthorDate: Mon Nov 15 12:05:32 2021 -0800
Add config for enabling realtime offset based consumption status checker
(#7753)
---
.../apache/pinot/common/utils/ServiceStatus.java| 21 -
.../server/starter/helix/BaseServerStarter.java | 16
.../org/apache/pinot/spi/utils/CommonConstants.java | 3 +++
3 files changed, 23 insertions(+), 17 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/ServiceStatus.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/ServiceStatus.java
index d50e399..45221fb 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/ServiceStatus.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/ServiceStatus.java
@@ -218,8 +218,6 @@ public class ServiceStatus {
private final Supplier
_getNumConsumingSegmentsNotReachedTheirLatestOffset;
String _statusDescription = STATUS_DESCRIPTION_INIT;
-private boolean _consumptionNotYetCaughtUp = true;
-
/**
* Realtime consumption catchup service which adds a static wait time for
consuming segments to catchup
*/
@@ -242,22 +240,19 @@ public class ServiceStatus {
return _serviceStatus;
}
long now = System.currentTimeMillis();
- int numConsumingSegmentsNotCaughtUp =
_getNumConsumingSegmentsNotReachedTheirLatestOffset.get();
+ boolean isConsumingSegmentsCounterProvided =
_getNumConsumingSegmentsNotReachedTheirLatestOffset != null;
+ int numConsumingSegmentsNotCaughtUp =
+ isConsumingSegmentsCounterProvided ?
_getNumConsumingSegmentsNotReachedTheirLatestOffset.get() : -1;
if (now >= _endWaitTime) {
_statusDescription = String.format("Consuming segments status GOOD
since %dms "
+ "(numConsumingSegmentsNotCaughtUp=%d)", _endWaitTime,
numConsumingSegmentsNotCaughtUp);
return Status.GOOD;
}
- if (_consumptionNotYetCaughtUp && numConsumingSegmentsNotCaughtUp > 0) {
-// TODO: Once the performance of offset based consumption checker is
validated:
-// - remove the log line
-// - uncomment the status & statusDescription lines
-// - remove variable _consumptionNotYetCaughtUp
-_consumptionNotYetCaughtUp = false;
-LOGGER.info("All consuming segments have reached their latest offsets!
"
-+ "Finished {} msec earlier than time threshold.", _endWaitTime -
now);
-// _statusDescription = "Consuming segments status GOOD as all consuming
segments have reached the latest offset";
-// return Status.GOOD;
+ if (isConsumingSegmentsCounterProvided &&
numConsumingSegmentsNotCaughtUp == 0) {
+_statusDescription = String.format(
+"Consuming segments status GOOD as all consuming segments have
reached the latest offset. "
++ "Finished %d msec earlier than time threshold.",
_endWaitTime - now);
+return Status.GOOD;
}
_statusDescription =
String.format("Waiting for consuming segments to catchup:
numConsumingSegmentsNotCaughtUp=%d, "
diff --git
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java
index fe010ba..aaa1e2a 100644
---
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java
+++
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/BaseServerStarter.java
@@ -29,6 +29,7 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.function.Supplier;
import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.helix.HelixAdmin;
@@ -221,6 +222,9 @@ public abstract class BaseServerStarter implements
ServiceStartable {
int realtimeConsumptionCatchupWaitMs = _serverConf
.getProperty(Server.CONFIG_OF_STARTUP_REALTIME_CONSUMPTION_CATCHUP_WAIT_MS,
Server.DEFAULT_STARTUP_REALTIME_CONSUMPTION_CATCHUP_WAIT_MS);
+boolean isOffsetBasedConsumptionStatusCheckerEnabled = _serverConf
+
.getProperty(Server.CONFIG_OF_ENABLE_REALTIME_OFFSET_BASED_CONSUMPTION_STATUS_CHECKER,
+
Server.DEFAULT_ENABLE_REALTIME_OFFSET_BASED_CONSUMPTION_STATUS_CHECKER);
// collect all resources which have this instance in the
[pinot] branch master updated: Use oldest offset on newly detected partitions (#7756)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 561b8a3 Use oldest offset on newly detected partitions (#7756)
561b8a3 is described below
commit 561b8a3c6fa832be3ab9e540ca3688a38a11e4c8
Author: Subbu Subramaniam
AuthorDate: Fri Nov 12 15:04:18 2021 -0800
Use oldest offset on newly detected partitions (#7756)
* Use oldest offset on newly detected partitions
Fixed the issue where Pinot was losing data when it detected new stream
partitions
(depending on table configuration).
Manual testing by running LLCRealtimeClusterIntegrationTest via debugger
- Change the table config to start from largest offset
- Force the test to detect only one partition, notice that roughly
half the rows are ingested, and only partiton 0 shows up in idealstate
- Run the RealtimeSegmentValidationManager job via swagger to force
detection
of new partition.
- Confirmed that all rows are now present.
Issue #7741
* Fixed linter error
---
.../core/realtime/PinotLLCRealtimeSegmentManager.java | 17 -
1 file changed, 12 insertions(+), 5 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
index 7f6996a..16c7471 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
@@ -316,7 +316,7 @@ public class PinotLLCRealtimeSegmentManager {
for (PartitionGroupMetadata partitionGroupMetadata :
newPartitionGroupMetadataList) {
String segmentName =
setupNewPartitionGroup(tableConfig, streamConfig,
partitionGroupMetadata, currentTimeMs, instancePartitions,
- numPartitionGroups, numReplicas, newPartitionGroupMetadataList);
+ numPartitionGroups, numReplicas, newPartitionGroupMetadataList,
false);
updateInstanceStatesForNewConsumingSegment(instanceStatesMap, null,
segmentName, segmentAssignment,
instancePartitionsMap);
@@ -1193,7 +1193,7 @@ public class PinotLLCRealtimeSegmentManager {
if (!latestSegmentZKMetadataMap.containsKey(partitionGroupId)) {
String newSegmentName =
setupNewPartitionGroup(tableConfig, streamConfig,
partitionGroupMetadata, currentTimeMs, instancePartitions,
-numPartitions, numReplicas, newPartitionGroupMetadataList);
+numPartitions, numReplicas, newPartitionGroupMetadataList,
true);
updateInstanceStatesForNewConsumingSegment(instanceStatesMap, null,
newSegmentName, segmentAssignment,
instancePartitionsMap);
}
@@ -1232,10 +1232,16 @@ public class PinotLLCRealtimeSegmentManager {
*/
private String setupNewPartitionGroup(TableConfig tableConfig,
PartitionLevelStreamConfig streamConfig,
PartitionGroupMetadata partitionGroupMetadata, long creationTimeMs,
InstancePartitions instancePartitions,
- int numPartitionGroups, int numReplicas, List
partitionGroupMetadataList) {
+ int numPartitionGroups, int numReplicas, List
partitionGroupMetadataList,
+ boolean isLiveTable) {
String realtimeTableName = tableConfig.getTableName();
int partitionGroupId = partitionGroupMetadata.getPartitionGroupId();
-String startOffset = partitionGroupMetadata.getStartOffset().toString();
+StreamPartitionMsgOffset startOffset;
+if (isLiveTable) {
+ startOffset = getPartitionGroupSmallestOffset(streamConfig,
partitionGroupId);
+} else {
+ startOffset = partitionGroupMetadata.getStartOffset();
+}
LOGGER.info("Setting up new partition group: {} for table: {}",
partitionGroupId, realtimeTableName);
String rawTableName =
TableNameBuilder.extractRawTableName(realtimeTableName);
@@ -1243,7 +1249,8 @@ public class PinotLLCRealtimeSegmentManager {
new LLCSegmentName(rawTableName, partitionGroupId,
STARTING_SEQUENCE_NUMBER, creationTimeMs);
String newSegmentName = newLLCSegmentName.getSegmentName();
-CommittingSegmentDescriptor committingSegmentDescriptor = new
CommittingSegmentDescriptor(null, startOffset, 0);
+CommittingSegmentDescriptor committingSegmentDescriptor = new
CommittingSegmentDescriptor(null,
+startOffset.toString(), 0);
createNewSegmentZKMetadata(tableConfig, streamConfig, newLLCSegmentName,
creationTimeMs,
committingSegmentDescriptor, null, instancePartitions,
numPartitionGroups, numReplicas,
partitionGroupMe
[pinot] branch master updated: fixing compatibility check for presto-pinot-driver (#7423)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new ffcf9b9 fixing compatibility check for presto-pinot-driver (#7423) ffcf9b9 is described below commit ffcf9b991431067c834bd4fb56fd7641c7fec172 Author: Xiang Fu AuthorDate: Sun Sep 12 09:53:24 2021 -0700 fixing compatibility check for presto-pinot-driver (#7423) --- pinot-connectors/presto-pinot-driver/pom.xml | 16 1 file changed, 16 insertions(+) diff --git a/pinot-connectors/presto-pinot-driver/pom.xml b/pinot-connectors/presto-pinot-driver/pom.xml index abedb03..3e8b29a 100644 --- a/pinot-connectors/presto-pinot-driver/pom.xml +++ b/pinot-connectors/presto-pinot-driver/pom.xml @@ -593,6 +593,22 @@ true + +maven-jar-plugin + +true + + + +default-jar +none + + +test-jar +none + + + - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated: Bug fix: make currentOffset in consuming segment volatile (#7365)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 0681e4b Bug fix: make currentOffset in consuming segment volatile
(#7365)
0681e4b is described below
commit 0681e4b55524ad316123eae40e7990a3e59c1781
Author: Sajjad Moradi
AuthorDate: Wed Aug 25 13:33:02 2021 -0700
Bug fix: make currentOffset in consuming segment volatile (#7365)
---
.../pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
index 422e11f..17ab449 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
@@ -227,7 +227,7 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
private final String _metricKeyName;
private final ServerMetrics _serverMetrics;
private final MutableSegmentImpl _realtimeSegment;
- private StreamPartitionMsgOffset _currentOffset;
+ private volatile StreamPartitionMsgOffset _currentOffset;
private volatile State _state;
private volatile int _numRowsConsumed = 0;
private volatile int _numRowsIndexed = 0; // Can be different from
_numRowsConsumed when metrics update is enabled.
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch master updated: Manually trigger PeriodicTask (#7174)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 7a44f4a Manually trigger PeriodicTask (#7174)
7a44f4a is described below
commit 7a44f4af2c5e2cb5dfb96c33d2010efb8e5f1e74
Author: Amrish Lal
AuthorDate: Mon Aug 23 12:06:14 2021 -0700
Manually trigger PeriodicTask (#7174)
* Don't allow a PeriodicTask to execute more than once at a time.
* Revert pom.xml change.
* Add controller message handler.
* Cleanup imports.
* Execute periodic task after controller receives message.
* Add test case.
* Additional log messages.
* Fix test case.
* Trigger task against specific table.
* Cleanup.
* Rebuild.
* Code review changes.
* Cleanup.
* Controller api to get list of available task names.
* Cleanup.
* Cleanup.
* Cleanup.
* Codereview changes.
* Cleanup.
* Controller API to set only one table and table type for manual periodic
task execution.
* Codereview changes.
* Cleanup.
* Codereview changes.
* Update test case to use countdown latch.
* Cleanup.
* Cleanup.
* Codreview changes.
* Add request id to link PeriodicTask execution reqeust with its log
entries.
* Cleanup.
* Rebuild.
* Fix NPE.
* Cleanup.
* Codereview changes.
* Cleanup.
* Codereview changes.
* Rebuild.
* Fix test case.
* Rebuild.
* Fix test case.
* Codereview changes.
* Cleanup.
* Cleanup /periodictask/run API
* Cleanup.
* Pass Properties object on the stack.
* Codereview changes.
* Fix checkstyle violations.
* fix checkstyle.
---
.../common/messages/RunPeriodicTaskMessage.java| 66 ++
.../pinot/controller/BaseControllerStarter.java| 6 +
...ControllerUserDefinedMessageHandlerFactory.java | 134 +
.../pinot/controller/api/resources/Constants.java | 1 +
...PinotControllerPeriodicTaskRestletResource.java | 123 +++
.../core/minion/MinionInstancesCleanupTask.java| 3 +-
.../helix/core/minion/TaskMetricsEmitter.java | 3 +-
.../core/periodictask/ControllerPeriodicTask.java | 26 +++-
.../minion/MinionInstancesCleanupTaskTest.java | 9 +-
.../pinot/core/periodictask/BasePeriodicTask.java | 93 --
.../pinot/core/periodictask/PeriodicTask.java | 11 ++
.../core/periodictask/PeriodicTaskScheduler.java | 62 +-
.../periodictask/PeriodicTaskSchedulerTest.java| 85 -
13 files changed, 574 insertions(+), 48 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/messages/RunPeriodicTaskMessage.java
b/pinot-common/src/main/java/org/apache/pinot/common/messages/RunPeriodicTaskMessage.java
new file mode 100644
index 000..11ee6cf
--- /dev/null
+++
b/pinot-common/src/main/java/org/apache/pinot/common/messages/RunPeriodicTaskMessage.java
@@ -0,0 +1,66 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.messages;
+
+import java.util.UUID;
+import org.apache.helix.ZNRecord;
+import org.apache.helix.model.Message;
+
+
+/**
+ * Upon receiving this message, Controller will execute the specified
PeriodicTask against the tables for which it is
+ * the lead controller. The message is sent whenever API call for executing a
PeriodicTask is invoked.
+ */
+public class RunPeriodicTaskMessage extends Message {
+ public static final String RUN_PERIODIC_TASK_MSG_SUB_TYPE =
"RUN_PERIODIC_TASK";
+ private static final String PERIODIC_TASK_REQUEST_ID = "requestId";
+ private static final String PERIODIC_TASK_NAME_KEY = "taskName";
+ private static final String TABLE_NAME_WITH_TYPE_KEY = "tableNameWithType";
+
+ /**
+ * @param taskRequest
[pinot] branch master updated: Fix maven settings file collision (#7287)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 46a4cf7 Fix maven settings file collision (#7287)
46a4cf7 is described below
commit 46a4cf7856a9ea7ac927f4386b08aeb19ba68e5c
Author: Subbu Subramaniam
AuthorDate: Thu Aug 12 09:02:03 2021 -0700
Fix maven settings file collision (#7287)
* Fix maven settings file collision
It is intended that a CI pipeline passes its own maven settings file
into the checkoutAndBuild command as an env variable rather than
creating a hard-coded setting file inside the script.
Creating a new settings file broke the CICD build pipeline in our
organization.
* Fix the path of the settings file
* Fix settings file to be outside source dir
* Output the entire build file on failure.
---
.../scripts/.pinot_compatibility_verifier.sh | 20 -
compatibility-verifier/checkoutAndBuild.sh | 25 --
2 files changed, 28 insertions(+), 17 deletions(-)
diff --git a/.github/workflows/scripts/.pinot_compatibility_verifier.sh
b/.github/workflows/scripts/.pinot_compatibility_verifier.sh
index d30fdc1..cbe76f6 100755
--- a/.github/workflows/scripts/.pinot_compatibility_verifier.sh
+++ b/.github/workflows/scripts/.pinot_compatibility_verifier.sh
@@ -27,6 +27,24 @@ netstat -i
df -h
+SETTINGS_FILE="../settings.xml"
+
+echo "http://maven.apache.org/SETTINGS/1.0.0\"";>
${SETTINGS_FILE}
+echo " xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"";>>
${SETTINGS_FILE}
+echo " xsi:schemaLocation=\"http://maven.apache.org/SETTINGS/1.0.0";>>
${SETTINGS_FILE}
+echo "
https://maven.apache.org/xsd/settings-1.0.0.xsd\";>">> ${SETTINGS_FILE}
+echo " ">> ${SETTINGS_FILE}
+echo "">> ${SETTINGS_FILE}
+echo " confluent-mirror">> ${SETTINGS_FILE}
+echo " confluent">> ${SETTINGS_FILE}
+echo " https://packages.confluent.io/maven/">> ${SETTINGS_FILE}
+echo " false">> ${SETTINGS_FILE}
+echo "">> ${SETTINGS_FILE}
+echo " ">> ${SETTINGS_FILE}
+echo "">> ${SETTINGS_FILE}
+
+# PINOT_MAVEN_OPTS is used to provide additional maven options to the
checkoutAndBuild.sh command
+export PINOT_MAVEN_OPTS="-s $(pwd)/${SETTINGS_FILE}"
compatibility-verifier/checkoutAndBuild.sh -w $WORKING_DIR -o $OLD_COMMIT
-compatibility-verifier/compCheck.sh -w $WORKING_DIR -t $TEST_SUITE
\ No newline at end of file
+compatibility-verifier/compCheck.sh -w $WORKING_DIR -t $TEST_SUITE
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
index d51968d..577b897 100755
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -75,19 +75,6 @@ function build() {
local versionOption="-Djdk.version=8"
mkdir -p ${MVN_CACHE_DIR}
- echo "http://maven.apache.org/SETTINGS/1.0.0\"";>
../settings.xml
- echo " xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"";>>
../settings.xml
- echo " xsi:schemaLocation=\"http://maven.apache.org/SETTINGS/1.0.0";>>
../settings.xml
- echo "
https://maven.apache.org/xsd/settings-1.0.0.xsd\";>">> ../settings.xml
- echo " ">> ../settings.xml
- echo "">> ../settings.xml
- echo " confluent-mirror">> ../settings.xml
- echo " confluent">> ../settings.xml
- echo " https://packages.confluent.io/maven/">>
../settings.xml
- echo " false">> ../settings.xml
- echo "">> ../settings.xml
- echo " ">> ../settings.xml
- echo "">> ../settings.xml
if [ ${buildId} -gt 0 ]; then
# Build it in a different env under different version so that maven cache
does
@@ -98,12 +85,12 @@ function build() {
repoOption="-Dmaven.repo.local=${mvnCache}/${buildId}"
fi
- mvn install package -s ../settings.xml -DskipTests -Pbin-dist
${versionOption} ${repoOption} ${PINOT_MAVEN_OPTS} 1>${outFile} 2>&1
+ mvn install package -DskipTests -Pbin-dist ${versionOption} ${repoOption}
${PINOT_MAVEN_OPTS} 1>${outFile} 2>&1
if [ $? -ne 0 ]; then exit 1; fi
- mvn -pl pinot-tools package -s ../settings.xml -DskipTests ${versionOption}
${repoOption} ${PINOT_MAVEN_OPTS} 1>>${outFile} 2>&1
+ mvn -pl pinot-tools package -DskipTests ${versionOption}
[pinot] branch master updated (385b3cf -> 6fb51d2)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git. from 385b3cf Update reflection lib from 0.9.11 to 0.9.9 (#7277) add 6fb51d2 Fix status return from compat checker (#7279) No new revisions were added by this update. Summary of changes: compatibility-verifier/checkoutAndBuild.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] branch fix-compat-incubator created (now c62c0cb)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch fix-compat-incubator in repository https://gitbox.apache.org/repos/asf/pinot.git. at c62c0cb Move pinot url away from incubator This branch includes the following new commits: new c62c0cb Move pinot url away from incubator The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[pinot] 01/01: Move pinot url away from incubator
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch fix-compat-incubator
in repository https://gitbox.apache.org/repos/asf/pinot.git
commit c62c0cbc0729acbd903a6b1778bcfa24eb66b392
Author: Subbu Subramaniam
AuthorDate: Fri Jul 23 14:40:43 2021 -0700
Move pinot url away from incubator
---
compatibility-verifier/checkoutAndBuild.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
index b536ff2..7f0ce44 100755
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -49,7 +49,7 @@ function checkOut() {
pushd "$targetDir" 1>&2 || exit 1
git init 1>&2 || exit 1
- git remote add origin https://github.com/apache/incubator-pinot 1>&2 || exit
1
+ git remote add origin https://github.com/apache/pinot 1>&2 || exit 1
git pull origin master 1>&2 || exit 1
# Pull the tag list so that we can check out by tag name
git fetch --tags 1>&2 || exit 1
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [7156] Human Readable Controller Configs (#7173)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 4c04e1f [7156] Human Readable Controller Configs (#7173)
4c04e1f is described below
commit 4c04e1f73798db2b739ae151c030ef2564be8367
Author: suddendust <84911643+suddend...@users.noreply.github.com>
AuthorDate: Fri Jul 23 21:39:37 2021 +0530
[7156] Human Readable Controller Configs (#7173)
* Initial commit:
1. Created period variables.
2. Modified getters of some 'inSeconds' variables to read the period config
and fallback to the 'inSeconds' config is the former is missing.
* Marked older configs as deprecated, added convertPeriodToUnit and
convertPeriodToSeconds
* Refactored multiple optionals code
* Added UT for ControllerConf.java
* Minor refactoring
* Removed DELETED_SEGMENTS_RETENTION_PERIOD to remove ambiguity in units
* Used correct method signature from getProperty(name, clazz) to
getProperty(name) to get properties of type strings
* Added UT to test when valid unit and invalid period configs are specified
* Addresses comments:
1. Rolled-back changes to controller.realtime.segment.commit.timeoutPeriod
and server.request.timeoutPeriod.
2. Corrected documentation of getters.
* Replaced List.of with Arrays.asList to make it compatible with JDK8
* spotless
* Addressed comments:
1. Prefixed DEPRECATED configs with DEPRECATED_
2. Handled disabling using -1 for controller.task.frequencyPeriod.
3. Un-did changes to controller.statuschecker.waitForPushTimePeriod.
4. Enhanced unit tests.
* Added license
* Removed changes to intial delay configs
* Removed unchanged configs from UT
* Fixed breaking integration test
* Marked deprecated configs with build version
* Addressed PR comments:
1. Removed DEPRECATED_VALIDATION_MANAGER_FREQUENCY_IN_SECONDS.
2. Printed the configuration under test for failing assertions.
---
.../apache/pinot/controller/ControllerConf.java| 263 +
.../pinot/controller/ControllerConfTest.java | 197 +++
.../ControllerPeriodicTasksIntegrationTest.java| 6 +-
.../UpsertTableSegmentUploadIntegrationTest.java | 4 +-
.../apache/pinot/tools/utils/PinotConfigUtils.java | 8 +-
5 files changed, 376 insertions(+), 102 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
index c2c93df..c388120 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
@@ -25,6 +25,7 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Random;
+import java.util.concurrent.TimeUnit;
import org.apache.commons.configuration.Configuration;
import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
@@ -75,45 +76,87 @@ public class ControllerConf extends PinotConfiguration {
public static class ControllerPeriodicTasksConf {
// frequency configs
-public static final String RETENTION_MANAGER_FREQUENCY_IN_SECONDS =
"controller.retention.frequencyInSeconds";
+// Deprecated as of 0.8.0
@Deprecated
-// The ValidationManager has been split up into 3 separate tasks, each
having their own frequency config settings
-public static final String
DEPRECATED_VALIDATION_MANAGER_FREQUENCY_IN_SECONDS =
-"controller.validation.frequencyInSeconds";
-public static final String
OFFLINE_SEGMENT_INTERVAL_CHECKER_FREQUENCY_IN_SECONDS =
+public static final String
DEPRECATED_RETENTION_MANAGER_FREQUENCY_IN_SECONDS =
+"controller.retention.frequencyInSeconds";
+public static final String RETENTION_MANAGER_FREQUENCY_PERIOD =
"controller.retention.frequencyPeriod";
+// Deprecated as of 0.8.0
+@Deprecated
+public static final String
DEPRECATED_OFFLINE_SEGMENT_INTERVAL_CHECKER_FREQUENCY_IN_SECONDS =
"controller.offline.segment.interval.checker.frequencyInSeconds";
-public static final String
REALTIME_SEGMENT_VALIDATION_FREQUENCY_IN_SECONDS =
+public static final String
OFFLINE_SEGMENT_INTERVAL_CHECKER_FREQUENCY_PERIOD =
+"controller.offline.segment.interval.checker.frequencyPeriod";
+// Deprecated as of 0.8.0
+@Deprecated
+public static final String
DEPRECATED_REALTIME_SEGMENT_VALIDATION_FREQUENCY_IN_SECONDS =
[incubator-pinot] branch master updated: Fix errors happening due to parallel builds (#7172)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new dbaac00 Fix errors happening due to parallel builds (#7172)
dbaac00 is described below
commit dbaac00b6f77161f079f07ff7d337cbde55d2cfd
Author: Subbu Subramaniam
AuthorDate: Sun Jul 18 10:09:01 2021 -0700
Fix errors happening due to parallel builds (#7172)
Maven cache interferes when we build two trees in parallel,
causing (for example) pinot-spi of one tree to be linked
with that of the other tree (since they are of the same version)
Changed the build to use different version ID for old and new builds
so that they don't pull in each other's jars. Current build always
uses the same ID as the current tree so that any changes made
can be tested easily without backing out pom files.
A different build ID each time can cause maven cache to explode if we
run the compile multiple times, so changed the build to use a dedicated
cache that is deleted after builds.
Also added code to cleanup the controller and server's data
directories before starting compat checker.
---
compatibility-verifier/checkoutAndBuild.sh | 48 +-
compatibility-verifier/compCheck.sh| 21 ++
.../org/apache/pinot/compat/tests/StreamOp.java| 11 -
3 files changed, 69 insertions(+), 11 deletions(-)
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
index 5e67128..b536ff2 100755
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -21,6 +21,7 @@
cmdName=`basename $0`
cmdDir=`dirname $0`
source ${cmdDir}/utils.inc
+MVN_CACHE_DIR="mvn-compat-cache"
# get usage of the script
function usage() {
@@ -60,21 +61,42 @@ function checkOut() {
# This function builds pinot code and tools necessary to run pinot-admin
commands
# Optionally, it also builds the integration test jars needed to run the pinot
# compatibility tester
+# It uses a different version for each build, since we build trees in parallel.
+# Building the same version on all trees in parallel causes unstable builds.
+# Using indpendent buildIds will cause maven cache to fill up, so we use a
+# dedicated maven cache for these builds, and remove the cache after build is
+# completed.
+# If buildId is less than 0, then the mvn version is not changed in pom files.
function build() {
local outFile=$1
local buildTests=$2
+ local buildId=$3
+ local repoOption=""
+ local versionOption="-Djdk.version=8"
- mvn install package -DskipTests -Pbin-dist -D jdk.version=8
${PINOT_MAVEN_OPTS} 1>${outFile} 2>&1
+ if [ ${buildId} -gt 0 ]; then
+# Build it in a different env under different version so that maven cache
does
+# not collide
+local pomVersion=$(grep -E "(.*)-SNAPSHOT" pom.xml |
cut -d'>' -f2 | cut -d'<' -f1 | cut -d'-' -f1)
+mvn versions:set -DnewVersion="${pomVersion}-compat-${buildId}" -q -B
1>${outFile} 2>&1
+mvn versions:commit -q -B 1>${outFile} 2>&1
+repoOption="-Dmaven.repo.local=${mvnCache}/${buildId}"
+ fi
+
+ mvn install package -DskipTests -Pbin-dist ${versionOption} ${repoOption}
${PINOT_MAVEN_OPTS} 1>${outFile} 2>&1
if [ $? -ne 0 ]; then exit 1; fi
- mvn -pl pinot-tools package -DskipTests -Djdk.version=8 ${PINOT_MAVEN_OPTS}
1>>${outFile} 2>&1
+ mvn -pl pinot-tools package -DskipTests ${versionOption} ${repoOption}
${PINOT_MAVEN_OPTS} 1>>${outFile} 2>&1
if [ $? -ne 0 ]; then exit 1; fi
if [ $buildTests -eq 1 ]; then
-mvn -pl pinot-integration-tests package -DskipTests -Djdk.version=8
${PINOT_MAVEN_OPTS} 1>>${outFile} 2>&1
+mvn -pl pinot-integration-tests package -DskipTests ${versionOption}
${repoOption} ${PINOT_MAVEN_OPTS} 1>>${outFile} 2>&1
if [ $? -ne 0 ]; then exit 1; fi
fi
-
}
+#
+# Main
+#
+
# get arguments
# Args while-loop
while [ "$1" != "" ]; do
@@ -128,6 +150,8 @@ oldTargetDir="$workingDir"/oldTargetDir
newBuildOutFile="${workingDir}/newBuild.out"
newTargetDir="$workingDir"/newTargetDir
curBuildOutFile="${workingDir}/currentBuild.out"
+mvnCache=${workingDir}/${MVN_CACHE_DIR}
+mkdir -p ${mvnCache}
# Find value of olderCommit hash
if [ -z "$olderCommit" -a ! -z "$newerCommit" ]; then
@@ -166,21 +190,23 @@ checkOut "$olderCommit" "$oldTargetDir"
# Start builds in parallel.
# First build the current tree. We need it so that we can
# run the compatibiity tester.
-echo Starting build for compat ch
[incubator-pinot] branch master updated (db5dfd7 -> de2f0e0)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from db5dfd7 Fix race condition in MinionInstancesCleanupTask (#7122) add de2f0e0 Enable parallel builds for compat checker (#7149) No new revisions were added by this update. Summary of changes: compatibility-verifier/checkoutAndBuild.sh | 194 ++--- 1 file changed, 148 insertions(+), 46 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Remove parallel builds in compat scripts (#7108)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 47a75e5 Remove parallel builds in compat scripts (#7108)
47a75e5 is described below
commit 47a75e5093129cc280de4c118434ccb337cd3da1
Author: Subbu Subramaniam
AuthorDate: Tue Jun 29 19:40:27 2021 -0700
Remove parallel builds in compat scripts (#7108)
Parallel builds seem to fail occasionally since some packages are not
thread-safe. We can always add back parallel builds vie PINOT_MAVEN_OPTS
if needed
---
compatibility-verifier/checkoutAndBuild.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
index f624cea..7d218dc 100755
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -52,7 +52,7 @@ function checkoutAndBuild() {
# Pull the tag list so that we can check out by tag name
git fetch --tags || exit 1
git checkout $commitHash || exit 1
- mvn install package -DskipTests -Pbin-dist -T 4 -Djdk.version=8
${PINOT_MAVEN_OPTS} || exit 1
+ mvn install package -DskipTests -Pbin-dist -Djdk.version=8
${PINOT_MAVEN_OPTS} || exit 1
popd || exit 1
exit 0
}
@@ -119,7 +119,7 @@ workingDir=$(absPath "$workingDir")
newTargetDir="$workingDir"/newTargetDir
if [ -z "$newerCommit" ]; then
echo "Compiling current tree as newer version"
- (cd $cmdDir/.. && mvn install package -DskipTests -Pbin-dist -T 4 -D
jdk.version=8 ${PINOT_MAVEN_OPTS} && mvn -pl pinot-tools package -T 4
-DskipTests -Djdk.version=8 ${PINOT_MAVEN_OPTS} && mvn -pl
pinot-integration-tests package -T 4 -DskipTests -Djdk.version=8
${PINOT_MAVEN_OPTS})
+ (cd $cmdDir/.. && mvn install package -DskipTests -Pbin-dist -D
jdk.version=8 ${PINOT_MAVEN_OPTS} && mvn -pl pinot-tools package -DskipTests
-Djdk.version=8 ${PINOT_MAVEN_OPTS} && mvn -pl pinot-integration-tests package
-DskipTests -Djdk.version=8 ${PINOT_MAVEN_OPTS})
if [ $? -ne 0 ]; then
echo Compile failed.
exit 1
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Fix maven options (#7105)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 2475089 Fix maven options (#7105)
2475089 is described below
commit 24750891e42641fa49a6c5f018318d7ffc7b6be6
Author: Subbu Subramaniam
AuthorDate: Tue Jun 29 17:22:12 2021 -0700
Fix maven options (#7105)
* Fix maven options
For some reason, MAVEN_OPTS seems to be triggering a build failure when
exported as an option. Changing the variable to PINOT_MAVEN_OPTS seems
to work.
* Fixed a typo
---
compatibility-verifier/checkoutAndBuild.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
index bc79422..f624cea 100755
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -36,7 +36,7 @@ function usage() {
echo -e "To compare this checkout with some older tag or hash:
'${cmdName} -o release-0.7.1 -w /tmp/wd'"
echo -e "To compare any two previous tags or hashes: '${cmdName} -o
release-0.7.1 -n 637cc3494 -w /tmp/wd"
echo -e "Environment:"
- echo -e "Additional maven build options can be passed in via environment
varibale MAVEN_OPTS"
+ echo -e "Additional maven build options can be passed in via environment
variable PINOT_MAVEN_OPTS"
exit 1
}
@@ -52,7 +52,7 @@ function checkoutAndBuild() {
# Pull the tag list so that we can check out by tag name
git fetch --tags || exit 1
git checkout $commitHash || exit 1
- mvn install package -DskipTests -Pbin-dist -T 4 -Djdk.version=8
${MAVEN_OPTS} || exit 1
+ mvn install package -DskipTests -Pbin-dist -T 4 -Djdk.version=8
${PINOT_MAVEN_OPTS} || exit 1
popd || exit 1
exit 0
}
@@ -119,7 +119,7 @@ workingDir=$(absPath "$workingDir")
newTargetDir="$workingDir"/newTargetDir
if [ -z "$newerCommit" ]; then
echo "Compiling current tree as newer version"
- (cd $cmdDir/.. && mvn install package -DskipTests -Pbin-dist -T 4 -D
jdk.version=8 ${MAVEN_OPTS} && mvn -pl pinot-tools package -T 4 -DskipTests
-Djdk.version=8 ${MAVEN_OPTS} && mvn -pl pinot-integration-tests package -T 4
-DskipTests -Djdk.version=8 ${MAVEN_OPTS})
+ (cd $cmdDir/.. && mvn install package -DskipTests -Pbin-dist -T 4 -D
jdk.version=8 ${PINOT_MAVEN_OPTS} && mvn -pl pinot-tools package -T 4
-DskipTests -Djdk.version=8 ${PINOT_MAVEN_OPTS} && mvn -pl
pinot-integration-tests package -T 4 -DskipTests -Djdk.version=8
${PINOT_MAVEN_OPTS})
if [ $? -ne 0 ]; then
echo Compile failed.
exit 1
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (bbdd615 -> 0981271)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from bbdd615 Added support for additional mvn options (#7103) add 0981271 Add minion metrics of task queueing time and task numbers (#7099) No new revisions were added by this update. Summary of changes: .../apache/pinot/minion/metrics/MinionGauge.java | 1 + .../pinot/minion/metrics/MinionQueryPhase.java | 3 +- .../minion/taskfactory/TaskFactoryRegistry.java| 40 +++--- 3 files changed, 31 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch fix-maven-opts updated (0d64694 -> 8364217)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch fix-maven-opts in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 0d64694 Fix maven options add 8364217 Fixed a typo No new revisions were added by this update. Summary of changes: compatibility-verifier/checkoutAndBuild.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch fix-maven-opts created (now 0d64694)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch fix-maven-opts in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 0d64694 Fix maven options No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Added support for additional mvn options (#7103)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new bbdd615 Added support for additional mvn options (#7103)
bbdd615 is described below
commit bbdd6151b8c6b596621addf1ae06e779c4009006
Author: Subbu Subramaniam
AuthorDate: Tue Jun 29 13:11:40 2021 -0700
Added support for additional mvn options (#7103)
Additional options (e.g. proxy settings) can be passed for maven build
commands
in firewalled environments
---
compatibility-verifier/checkoutAndBuild.sh | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
index 47eb4ae..bc79422 100755
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -35,6 +35,8 @@ function usage() {
echo -e "To compare this checkout with previous commit: '${cmdName} -w
/tmp/wd'"
echo -e "To compare this checkout with some older tag or hash:
'${cmdName} -o release-0.7.1 -w /tmp/wd'"
echo -e "To compare any two previous tags or hashes: '${cmdName} -o
release-0.7.1 -n 637cc3494 -w /tmp/wd"
+ echo -e "Environment:"
+ echo -e "Additional maven build options can be passed in via environment
varibale MAVEN_OPTS"
exit 1
}
@@ -50,7 +52,7 @@ function checkoutAndBuild() {
# Pull the tag list so that we can check out by tag name
git fetch --tags || exit 1
git checkout $commitHash || exit 1
- mvn install package -DskipTests -Pbin-dist -T 4 -Djdk.version=8 || exit 1
+ mvn install package -DskipTests -Pbin-dist -T 4 -Djdk.version=8
${MAVEN_OPTS} || exit 1
popd || exit 1
exit 0
}
@@ -117,7 +119,7 @@ workingDir=$(absPath "$workingDir")
newTargetDir="$workingDir"/newTargetDir
if [ -z "$newerCommit" ]; then
echo "Compiling current tree as newer version"
- (cd $cmdDir/.. && mvn install package -DskipTests -Pbin-dist -T 4 -D
jdk.version=8 && mvn -pl pinot-tools package -T 4 -DskipTests -Djdk.version=8
&& mvn -pl pinot-integration-tests package -T 4 -DskipTests -Djdk.version=8)
+ (cd $cmdDir/.. && mvn install package -DskipTests -Pbin-dist -T 4 -D
jdk.version=8 ${MAVEN_OPTS} && mvn -pl pinot-tools package -T 4 -DskipTests
-Djdk.version=8 ${MAVEN_OPTS} && mvn -pl pinot-integration-tests package -T 4
-DskipTests -Djdk.version=8 ${MAVEN_OPTS})
if [ $? -ne 0 ]; then
echo Compile failed.
exit 1
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Added TaskMetricsEmitted periodic controller job (#7091)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new c950926 Added TaskMetricsEmitted periodic controller job (#7091)
c950926 is described below
commit c9509261672d0f9c2b35fa834eb8fe998ed23161
Author: Subbu Subramaniam
AuthorDate: Mon Jun 28 14:01:14 2021 -0700
Added TaskMetricsEmitted periodic controller job (#7091)
* Added TaskMetricsEmitted periodic controller job
This job runs every 5 mins by default and emits metrics about Minion tasks
scheduled in Pinot. For now, the following metrics are emitted for each
task type:
- Number of running tasks
- Number of running sub-tasks
- Number of waiting sub-taks (unassigned to a minion as yet)
- Number of error sub-tasks (completed with an error/exception)
- Percent of sub-tasks in error
- Percent of sub-tasks in waiting or running states.
* Addressed review comments and added unit tests
* Addressed review comments and resolved conflicts
* Cleaned up with spotless:apply
* Handle another null case
* spotless apply
* Fix compile error after merge
* Added comments on the minion integration tests
---
.../pinot/common/metrics/ControllerGauge.java | 6 ++
.../apache/pinot/controller/ControllerConf.java| 12 +++
.../core/minion/PinotHelixTaskResourceManager.java | 118 -
.../helix/core/minion/PinotTaskManager.java| 3 +-
.../helix/core/minion/TaskMetricsEmitter.java | 90
.../tests/SimpleMinionClusterIntegrationTest.java | 36 +--
6 files changed, 253 insertions(+), 12 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java
index 1ebeac1..2b91ba0 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerGauge.java
@@ -45,6 +45,12 @@ public enum ControllerGauge implements AbstractMetrics.Gauge
{
OFFLINE_TABLE_COUNT("TableCount", true),
DISABLED_TABLE_COUNT("TableCount", true),
PERIODIC_TASK_NUM_TABLES_PROCESSED("PeriodicTaskNumTablesProcessed", true),
+ NUM_MINION_TASKS_IN_PROGRESS("NumMinionTasksInProgress", true),
+ NUM_MINION_SUBTASKS_WAITING("NumMinionSubtasksWaiting", true),
+ NUM_MINION_SUBTASKS_RUNNING("NumMinionSubtasksRunning", true),
+ NUM_MINION_SUBTASKS_ERROR("NumMinionSubtasksError", true),
+ PERCENT_MINION_SUBTASKS_IN_QUEUE("PercentMinionSubtasksInQueue", true),
+ PERCENT_MINION_SUBTASKS_IN_ERROR("PercentMinionSubtasksInError", true),
// Pinot controller leader
PINOT_CONTROLLER_LEADER("PinotControllerLeader", true),
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
index 80bd173..7db3a27 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
@@ -98,6 +98,7 @@ public class ControllerConf extends PinotConfiguration {
"controller.minion.instances.cleanup.task.frequencyInSeconds";
public static final String
MINION_INSTANCES_CLEANUP_TASK_INITIAL_DELAY_SECONDS =
"controller.minion.instances.cleanup.task.initialDelaySeconds";
+public static final String TASK_METRICS_EMITTER_FREQUENCY_IN_SECONDS =
"controller.minion.task.metrics.emitter.frequencyInSeconds";
public static final String PINOT_TASK_MANAGER_SCHEDULER_ENABLED =
"controller.task.scheduler.enabled";
@Deprecated
@@ -140,6 +141,7 @@ public class ControllerConf extends PinotConfiguration {
private static final int
DEFAULT_REALTIME_SEGMENT_VALIDATION_FREQUENCY_IN_SECONDS = 60 * 60; // 1 Hour.
private static final int
DEFAULT_BROKER_RESOURCE_VALIDATION_FREQUENCY_IN_SECONDS = 60 * 60; // 1 Hour.
private static final int DEFAULT_STATUS_CONTROLLER_FREQUENCY_IN_SECONDS =
5 * 60; // 5 minutes
+private static final int DEFAULT_TASK_METRICS_EMITTER_FREQUENCY_IN_SECONDS
= 5 * 60; // 5 minutes
private static final int
DEFAULT_STATUS_CONTROLLER_WAIT_FOR_PUSH_TIME_IN_SECONDS = 10 * 60; // 10 minutes
private static final int DEFAULT_TASK_MANAGER_FREQUENCY_IN_SECONDS = -1;
// Disabled
private static final int
DEFAULT_MINION_INSTANCES_CLEANUP_TASK_FREQUENCY_IN_SECONDS = 60 * 60; // 1 Hour.
@@ -490,6 +492,16 @@ public class ControllerConf extends PinotConfiguration {
I
[incubator-pinot] 01/01: Fixed a comment
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch minion-improvements
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit a9f9e8aab23f5ff9ecbda91b9db5d723e2c6d912
Author: Subbu Subramaniam
AuthorDate: Mon Jun 28 10:05:15 2021 -0700
Fixed a comment
---
.../pinot/integration/tests/SimpleMinionClusterIntegrationTest.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/SimpleMinionClusterIntegrationTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/SimpleMinionClusterIntegrationTest.java
index 930a41e..bcdf6a5 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/SimpleMinionClusterIntegrationTest.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/SimpleMinionClusterIntegrationTest.java
@@ -121,8 +121,8 @@ public class SimpleMinionClusterIntegrationTest extends
ClusterTest {
verifyTaskCount(task2, 0, 2, 0, 2);
// Should not generate more tasks since
SimpleMinionClusterIntegrationTests.NUM_TASKS is 2.
-// Our test task generator does not generate if there are already this
many sub-tasks in the
-// running+waiting count already.
+// Our test task generator does not generate if there are already this
many top-level tasks
+// in progress.
assertNull(_taskManager.scheduleTasks().get(TASK_TYPE));
assertNull(_taskManager.scheduleTask(TASK_TYPE));
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch minion-improvements created (now a9f9e8a)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch minion-improvements in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at a9f9e8a Fixed a comment This branch includes the following new commits: new a9f9e8a Fixed a comment The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (9e26c24 -> 282edba)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 9e26c24 Add support for range index rule recommendation(#7034) (#7063) add 282edba Adding support for config files in compat tests (#7074) No new revisions were added by this update. Summary of changes: compatibility-verifier/checkoutAndBuild.sh | 4 +- compatibility-verifier/compCheck.sh| 76 +- .../config/BrokerConfig.properties | 6 +- .../config/ControllerConfig.properties | 8 ++- .../config/ServerConfig.properties | 9 ++- .../pinot/compat/tests/ClusterDescriptor.java | 64 +++--- .../pinot/compat/tests/CompatibilityOpsRunner.java | 4 ++ .../org/apache/pinot/compat/tests/QueryOp.java | 2 +- .../org/apache/pinot/compat/tests/SegmentOp.java | 13 ++-- .../org/apache/pinot/compat/tests/StreamOp.java| 7 +- .../org/apache/pinot/compat/tests/TableOp.java | 9 ++- 11 files changed, 152 insertions(+), 50 deletions(-) copy docker/images/pinot-superset/.dockerignore => compatibility-verifier/sample-test-suite/config/BrokerConfig.properties (88%) copy docker/images/pinot-presto/etc/catalog/pinot.properties => compatibility-verifier/sample-test-suite/config/ControllerConfig.properties (81%) copy docker/images/pinot-presto/etc/catalog/pinot.properties => compatibility-verifier/sample-test-suite/config/ServerConfig.properties (75%) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch add-config-file-support updated (6eba9df -> 8fbb4aa)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch add-config-file-support in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 6eba9df Adding support for config files in compat tests add 8fbb4aa Addressed review comments No new revisions were added by this update. Summary of changes: .../pinot/compat/tests/ClusterDescriptor.java | 30 +- .../pinot/compat/tests/CompatibilityOpsRunner.java | 8 +++--- .../org/apache/pinot/compat/tests/QueryOp.java | 2 +- .../org/apache/pinot/compat/tests/SegmentOp.java | 13 ++ .../org/apache/pinot/compat/tests/StreamOp.java| 7 ++--- .../org/apache/pinot/compat/tests/TableOp.java | 9 --- 6 files changed, 41 insertions(+), 28 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch add-config-file-support created (now 6eba9df)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch add-config-file-support in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 6eba9df Adding support for config files in compat tests This branch includes the following new commits: new 6eba9df Adding support for config files in compat tests The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Adding support for config files in compat tests
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch add-config-file-support
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 6eba9df5b3271aee885ddf7bac6fb32b55e164a7
Author: Subbu Subramaniam
AuthorDate: Fri Jun 18 15:33:10 2021 -0700
Adding support for config files in compat tests
Pinot installation administrators can now specify configuration files to
run the
tests as per their environment.
Changed the java files to get the ports as declared in the config files.
Cleaned up the scripts to use variables for ports.
Updated the scripts to compile with java 8 (so that it can be run on
desktops
with one java environment for older and newer release)
---
compatibility-verifier/checkoutAndBuild.sh | 4 +-
compatibility-verifier/compCheck.sh| 76 +-
.../config/BrokerConfig.properties | 22 +++
.../config/ControllerConfig.properties | 24 +++
.../config/ServerConfig.properties | 25 +++
.../pinot/compat/tests/ClusterDescriptor.java | 58 ++---
.../pinot/compat/tests/CompatibilityOpsRunner.java | 4 ++
.../org/apache/pinot/compat/tests/QueryOp.java | 2 +-
.../org/apache/pinot/compat/tests/SegmentOp.java | 10 +--
.../org/apache/pinot/compat/tests/StreamOp.java| 6 +-
.../org/apache/pinot/compat/tests/TableOp.java | 6 +-
11 files changed, 196 insertions(+), 41 deletions(-)
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
index 36d9edf..47eb4ae 100755
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -50,7 +50,7 @@ function checkoutAndBuild() {
# Pull the tag list so that we can check out by tag name
git fetch --tags || exit 1
git checkout $commitHash || exit 1
- mvn install package -DskipTests -Pbin-dist || exit 1
+ mvn install package -DskipTests -Pbin-dist -T 4 -Djdk.version=8 || exit 1
popd || exit 1
exit 0
}
@@ -117,7 +117,7 @@ workingDir=$(absPath "$workingDir")
newTargetDir="$workingDir"/newTargetDir
if [ -z "$newerCommit" ]; then
echo "Compiling current tree as newer version"
- (cd $cmdDir/.. && mvn install package -DskipTests -Pbin-dist && mvn -pl
pinot-tools package -DskipTests && mvn -pl pinot-integration-tests package
-DskipTests)
+ (cd $cmdDir/.. && mvn install package -DskipTests -Pbin-dist -T 4 -D
jdk.version=8 && mvn -pl pinot-tools package -T 4 -DskipTests -Djdk.version=8
&& mvn -pl pinot-integration-tests package -T 4 -DskipTests -Djdk.version=8)
if [ $? -ne 0 ]; then
echo Compile failed.
exit 1
diff --git a/compatibility-verifier/compCheck.sh
b/compatibility-verifier/compCheck.sh
index 9d6cc97..3f77cdf 100755
--- a/compatibility-verifier/compCheck.sh
+++ b/compatibility-verifier/compCheck.sh
@@ -60,19 +60,18 @@ function waitForZkReady() {
status=1
while [ $status -ne 0 ]; do
sleep 1
-echo Checking port 2181 for zk ready
-echo x | nc localhost 2181 1>/dev/null 2>&1
+echo Checking port ${ZK_PORT} for zk ready
+echo x | nc localhost ${ZK_PORT} 1>/dev/null 2>&1
status=$(echo $?)
done
}
function waitForControllerReady() {
- # TODO: Check real controller port if config file is specified.
status=1
while [ $status -ne 0 ]; do
sleep 1
-echo Checking port 9000 for controller ready
-curl localhost:9000/health 1>/dev/null 2>&1
+echo Checking port ${CONTROLLER_PORT} for controller ready
+curl localhost:${CONTROLLER_PORT}/health 1>/dev/null 2>&1
status=$(echo $?)
done
}
@@ -88,23 +87,21 @@ function waitForKafkaReady() {
}
function waitForBrokerReady() {
- # TODO: We are checking for port 8099. Check for real broker port from
config if needed.
local status=1
while [ $status -ne 0 ]; do
sleep 1
-echo Checking port 8099 for broker ready
-curl localhost:8099/debug/routingTable 1>/dev/null 2>&1
+echo Checking port ${BROKER_QUERY_PORT} for broker ready
+curl localhost:${BROKER_QUERY_PORT}/debug/routingTable 1>/dev/null 2>&1
status=$(echo $?)
done
}
function waitForServerReady() {
- # TODO: We are checking for port 8097. Check for real server port from
config if needed,
local status=1
while [ $status -ne 0 ]; do
sleep 1
-echo Checking port 8097 for server ready
-curl localhost:8097/health 1>/dev/null 2>&1
+echo Checking port ${SERVER_ADMIN_PORT} for server ready
+curl localhost:${SERVER_ADMIN_PORT}/health 1>/dev/null 2>&1
status=$(echo $?)
done
}
@@ -150,7 +147,7 @@ function startService() {
./pinot-admin.sh StartServer ${configFileArg}
1>${LOG_DIR}/server.${logCount}.log 2>&1 &am
[incubator-pinot] branch master updated (a1c9b63 -> e91f426)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from a1c9b63 Produce GenericRow file in segment processing mapper (#7013) add e91f426 Improvements in compat-tester (#7024) No new revisions were added by this update. Summary of changes: compatibility-verifier/README.md | 17 - compatibility-verifier/checkoutAndBuild.sh | 109 ++--- compatibility-verifier/compCheck.sh| 14 ++-- 3 files changed, 91 insertions(+), 49 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch improv-compat created (now 401fa5c)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch improv-compat in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 401fa5c Updated README.md This branch includes the following new commits: new 401fa5c Updated README.md The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Updated README.md
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a commit to branch improv-compat in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 401fa5cd4617b21e57103f106c1479eb63a6a49a Author: Subbu Subramaniam AuthorDate: Sat Jun 5 19:12:44 2021 -0700 Updated README.md --- compatibility-verifier/README.md | 17 ++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/compatibility-verifier/README.md b/compatibility-verifier/README.md index 1c7090b..209ae09 100644 --- a/compatibility-verifier/README.md +++ b/compatibility-verifier/README.md @@ -25,10 +25,21 @@ ### Step 1: checkout source code and build targets for older commit and newer commit ```shell -./compatibility-verifier/checkoutAndBuild.sh -o olderCommit -n newerCommit [-w workingDir] +Usage: checkoutAndBuild.sh [-o olderCommit] [-n newerCommit] -w workingDir + -w, --working-dir Working directory where olderCommit and newCommit target files reside + + -o, --old-commit-hash git hash (or tag) for old commit + + -n, --new-commit-hash git hash (or tag) for new commit + +If -n is not specified, then current commit is assumed +If -o is not specified, then previous commit is assumed (expected -n is also empty) +Examples: +To compare this checkout with previous commit: 'checkoutAndBuild.sh -w /tmp/wd' +To compare this checkout with some older tag or hash: 'checkoutAndBuild.sh -o release-0.7.1 -w /tmp/wd' +To compare any two previous tags or hashes: 'checkoutAndBuild.sh -o release-0.7.1 -n 637cc3494 -w /tmp/wd + ``` -***NOTE***: `[workingDir]` is optional, if user does not specify `[workingDir]`, the script will create a temporary working -dir and output the path, which can be used in step 2. ### Step 2: run compatibility regression test against the two targets build in step1 ```shell - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Changed compat tester to make yaml files optional (#7020)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 0b14b96 Changed compat tester to make yaml files optional (#7020) 0b14b96 is described below commit 0b14b962be95b28fe28adc571106be3db9fdbbbe Author: Subbu Subramaniam AuthorDate: Fri Jun 4 09:53:23 2021 -0700 Changed compat tester to make yaml files optional (#7020) A test suite may choose to specify test operations for some stages and not others. Also fixed a typo in "basename" --- compatibility-verifier/compCheck.sh | 116 +++- 1 file changed, 73 insertions(+), 43 deletions(-) diff --git a/compatibility-verifier/compCheck.sh b/compatibility-verifier/compCheck.sh index 7d4fa1b..73af445 100755 --- a/compatibility-verifier/compCheck.sh +++ b/compatibility-verifier/compCheck.sh @@ -42,7 +42,7 @@ RM="/bin/rm" logCount=1 #Declare the number of mandatory args -cmdName=`baseName $0` +cmdName=`basename $0` source `dirname $0`/utils.inc # get usage of the script @@ -287,51 +287,69 @@ fi # Setup initial cluster with olderCommit and do rolling upgrade # Provide abspath of filepath to $COMPAT_TESTER +echo "Setting up cluster before upgrade" startServices "$oldTargetDir" -echo "Setting up cluster before upgrade" -$COMPAT_TESTER $testSuiteDir/pre-controller-upgrade.yaml 1 -if [ $? -ne 0 ]; then - if [ $keepClusterOnFailure == "false" ]; then -stopServices +genNum=0 +if [ -f $testSuiteDir/pre-controller-upgrade.yaml ]; then + genNum=$((genNum+1)) + $COMPAT_TESTER $testSuiteDir/pre-controller-upgrade.yaml $genNum + if [ $? -ne 0 ]; then +echo Failed before controller upgrade +if [ $keepClusterOnFailure == "false" ]; then + stopServices +fi +exit 1 fi - exit 1 fi echo "Upgrading controller" stopService controller startService controller "$newTargetDir" "$CONTROLLER_CONF" waitForControllerReady -echo "Running tests after controller upgrade" -$COMPAT_TESTER $testSuiteDir/pre-broker-upgrade.yaml 2 -if [ $? -ne 0 ]; then - if [ $keepClusterOnFailure == "false" ]; then -stopServices + +if [ -f $testSuiteDir/pre-broker-upgrade.yaml ]; then + genNum=$((genNum+1)) + echo "Running tests after controller upgrade" + $COMPAT_TESTER $testSuiteDir/pre-broker-upgrade.yaml $genNum + if [ $? -ne 0 ]; then +echo Failed before broker upgrade +if [ $keepClusterOnFailure == "false" ]; then + stopServices +fi +exit 1 fi - exit 1 fi echo "Upgrading broker" stopService broker startService broker "$newTargetDir" "$BROKER_CONF" waitForBrokerReady -echo "Running tests after broker upgrade" -$COMPAT_TESTER $testSuiteDir/pre-server-upgrade.yaml 3 -if [ $? -ne 0 ]; then - if [ $keepClusterOnFailure == "false" ]; then -stopServices +if [ -f $testSuiteDir/pre-server-upgrade.yaml ]; then + echo "Running tests after broker upgrade" + genNum=$((genNum+1)) + $COMPAT_TESTER $testSuiteDir/pre-server-upgrade.yaml $genNum + if [ $? -ne 0 ]; then +echo Failed before server upgrade +if [ $keepClusterOnFailure == "false" ]; then + stopServices +fi +exit 1 fi - exit 1 fi echo "Upgrading server" stopService server startService server "$newTargetDir" "$SERVER_CONF" waitForServerReady -echo "Running tests after server upgrade" -$COMPAT_TESTER $testSuiteDir/post-server-upgrade.yaml 4 -if [ $? -ne 0 ]; then - if [ $keepClusterOnFailure == "false" ]; then -stopServices +if [ -f $testSuiteDir/post-server-upgrade.yaml ]; then + echo "Running tests after server upgrade" + genNum=$((genNum+1)) + $COMPAT_TESTER $testSuiteDir/post-server-upgrade.yaml $genNum + if [ $? -ne 0 ]; then +echo Failed after server upgrade +if [ $keepClusterOnFailure == "false" ]; then + stopServices +fi +exit 1 fi - exit 1 fi echo "Downgrading server" @@ -339,38 +357,50 @@ echo "Downgrading server" stopService server startService server "$oldTargetDir" "$SERVER_CONF" waitForServerReady -echo "Running tests after server downgrade" -$COMPAT_TESTER $testSuiteDir/post-server-rollback.yaml 5 -if [ $? -ne 0 ]; then - if [ $keepClusterOnFailure == "false" ]; then -stopServices +if [ -f $testSuiteDir/post-server-rollback.yaml ]; then + echo "Running tests after server downgrade" + genNum=$((genNum+1)) + $COMPAT_TESTER $testSuiteDir/post-server-rollback.yaml $genNum + if [ $? -ne 0 ]; then +echo Failed after server downgrade +if [ $keepClusterOnFailure ==
[incubator-pinot] branch master updated: Resolve bug in HelixServerStarter for populating environment property map field and remame timeout variables to follow timeunit convention (#7019)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 1d1bf4d Resolve bug in HelixServerStarter for populating environment
property map field and remame timeout variables to follow timeunit convention
(#7019)
1d1bf4d is described below
commit 1d1bf4d96785a1011066f6ee897fc7985b31686c
Author: Jay Desai
AuthorDate: Thu Jun 3 17:33:22 2021 -0700
Resolve bug in HelixServerStarter for populating environment property map
field and remame timeout variables to follow timeunit convention (#7019)
Co-authored-by: Jay Desai
---
.../pinot/plugin/provider/AzureEnvironmentProvider.java | 16 ++--
.../pinot/server/starter/helix/HelixServerStarter.java | 2 +-
.../PinotEnvironmentProviderFactoryTest.java | 8
3 files changed, 11 insertions(+), 15 deletions(-)
diff --git
a/pinot-plugins/pinot-environment/pinot-azure/src/main/java/org/apache/pinot/plugin/provider/AzureEnvironmentProvider.java
b/pinot-plugins/pinot-environment/pinot-azure/src/main/java/org/apache/pinot/plugin/provider/AzureEnvironmentProvider.java
index c710ae8..c9c59d5 100644
---
a/pinot-plugins/pinot-environment/pinot-azure/src/main/java/org/apache/pinot/plugin/provider/AzureEnvironmentProvider.java
+++
b/pinot-plugins/pinot-environment/pinot-azure/src/main/java/org/apache/pinot/plugin/provider/AzureEnvironmentProvider.java
@@ -40,8 +40,6 @@ import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.apache.pinot.spi.env.PinotConfiguration;
import org.apache.pinot.spi.environmentprovider.PinotEnvironmentProvider;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
/**
@@ -49,12 +47,10 @@ import org.slf4j.LoggerFactory;
*/
public class AzureEnvironmentProvider implements PinotEnvironmentProvider {
- private static final Logger LOGGER =
LoggerFactory.getLogger(AzureEnvironmentProvider.class);
-
protected static final String MAX_RETRY = "maxRetry";
protected static final String IMDS_ENDPOINT = "imdsEndpoint";
- protected static final String CONNECTION_TIMEOUT = "connectionTimeout";
- protected static final String REQUEST_TIMEOUT = "requestTimeout";
+ protected static final String CONNECTION_TIMEOUT_MILLIS =
"connectionTimeoutMillis";
+ protected static final String REQUEST_TIMEOUT_MILLIS =
"requestTimeoutMillis";
private static final String COMPUTE = "compute";
private static final String METADATA = "Metadata";
private static final String PLATFORM_FAULT_DOMAIN = "platformFaultDomain";
@@ -73,12 +69,12 @@ public class AzureEnvironmentProvider implements
PinotEnvironmentProvider {
_maxRetry = Integer.parseInt(pinotConfiguration.getProperty(MAX_RETRY));
_imdsEndpoint = pinotConfiguration.getProperty(IMDS_ENDPOINT);
-int connectionTimeout =
Integer.parseInt(pinotConfiguration.getProperty(CONNECTION_TIMEOUT));
-int requestTimeout =
Integer.parseInt(pinotConfiguration.getProperty(REQUEST_TIMEOUT));
+int connectionTimeoutMillis =
Integer.parseInt(pinotConfiguration.getProperty(CONNECTION_TIMEOUT_MILLIS));
+int requestTimeoutMillis =
Integer.parseInt(pinotConfiguration.getProperty(REQUEST_TIMEOUT_MILLIS));
final RequestConfig requestConfig = RequestConfig.custom()
-.setConnectTimeout(connectionTimeout)
-.setConnectionRequestTimeout(requestTimeout)
+.setConnectTimeout(connectionTimeoutMillis)
+.setConnectionRequestTimeout(requestTimeoutMillis)
.build();
final HttpRequestRetryHandler httpRequestRetryHandler = (iOException,
executionCount, httpContext) ->
diff --git
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
index 1a154e4..ef239fe 100644
---
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
+++
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
@@ -303,7 +303,7 @@ public class HelixServerStarter implements ServiceStartable
{
Map existingEnvironmentConfigsMap =
instanceConfig.getRecord().getMapField(
CommonConstants.ENVIRONMENT_IDENTIFIER);
- if (existingEnvironmentConfigsMap != null &&
!existingEnvironmentConfigsMap.equals(environmentProperties)) {
+ if (existingEnvironmentConfigsMap == null ||
!existingEnvironmentConfigsMap.equals(environmentProperties)) {
instanceConfig.getRecord().setMapField(CommonConstants.ENVIRONMENT_IDENTIFIER,
environmentProperties);
LOGGER.info("Adding environment properties: {} for instance: {}",
environmentProperties, _
[incubator-pinot] branch master updated (fdef9bd -> c859044)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from fdef9bd Extract duplicate code in testSelectionOrderByAgoFunction into a single private helper function (#7012) add c859044 Modified the checkout scripts to parse arguments (#7011) No new revisions were added by this update. Summary of changes: compatibility-verifier/README.md | 2 +- compatibility-verifier/checkoutAndBuild.sh | 62 ++ compatibility-verifier/compCheck.sh| 60 - .../utils.inc | 14 +++-- 4 files changed, 73 insertions(+), 65 deletions(-) copy docker/images/pinot-presto/etc/catalog/pinot_quickstart.properties => compatibility-verifier/utils.inc (77%) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (9040571 -> 02d54b9)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 9040571 Fix partial comparison for realtime table and allow keep cluster on failure (#6969) add 02d54b9 Fix broker log: keep all the pairs together, and the query at the end (#7003) No new revisions were added by this update. Summary of changes: .../pinot/broker/requesthandler/BaseBrokerRequestHandler.java | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (79507da -> 9040571)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 79507da Enhance debug endpoint for table debugging. (#6977) add 9040571 Fix partial comparison for realtime table and allow keep cluster on failure (#6969) No new revisions were added by this update. Summary of changes: compatibility-verifier/README.md | 12 +- compatibility-verifier/compCheck.sh| 171 ++ .../queries/feature-test-2-sql-realtime.queries| 4 + .../feature-test-2-sql-realtime.results| 3 +- .../pinot/compat/tests/SqlResultComparator.java| 263 ++--- 5 files changed, 375 insertions(+), 78 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Enabling compatibility tests in the script (#6959)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 958b4d4 Enabling compatibility tests in the script (#6959)
958b4d4 is described below
commit 958b4d4e0a50685a5a95a149144e9f4ac765e7cb
Author: Subbu Subramaniam
AuthorDate: Mon May 24 11:01:01 2021 -0700
Enabling compatibility tests in the script (#6959)
The actual run of compat tests were disahled in the script. This PR
enables the compat tests.
Fixed the verifier script to work with relative as well as absolute
paths for working directory.
Other improvements:
We now preserve the log files of all processes we start.
Changed all pid files to be in one dir so it is easy to stop the cluster
Changed the mechanism to check if some service is up. This part needs more
improvement.
As yet, the tests fail after server upgrade
---
compatibility-verifier/checkoutAndBuild.sh | 4 +-
compatibility-verifier/compCheck.sh| 220 +++--
.../org/apache/pinot/compat/tests/SegmentOp.java | 4 +-
3 files changed, 168 insertions(+), 60 deletions(-)
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
old mode 100644
new mode 100755
index 5d71c21..ec3d7c6
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -37,8 +37,8 @@ function checkoutAndBuild() {
pushd "$targetDir" || exit 1
git init
git remote add origin https://github.com/apache/incubator-pinot
- git fetch --depth 1 origin "$commitHash"
- git checkout FETCH_HEAD
+ git pull origin master
+ git checkout $commitHash
mvn install package -DskipTests -Pbin-dist
popd || exit 1
}
diff --git a/compatibility-verifier/compCheck.sh
b/compatibility-verifier/compCheck.sh
index 6df83a2..0674c0e 100755
--- a/compatibility-verifier/compCheck.sh
+++ b/compatibility-verifier/compCheck.sh
@@ -39,6 +39,8 @@
# one more segment to table, adding some more rows to the stream topic, and
# running some queries with the new data.
+RM="/bin/rm"
+logCount=1
# get usage of the script
function usage() {
@@ -48,27 +50,62 @@ function usage() {
}
function waitForZkReady() {
- # TODO: check ZK to be ready instead of sleep
- sleep 60
- echo "zookeeper is ready"
+ status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 2181 for zk ready
+echo x |nc localhost 2181 1>/dev/null 2>&1
+status=$(echo $?)
+ done
}
function waitForControllerReady() {
- # TODO: check Controller to be ready instead of sleep
- sleep 60
- echo "controller is ready"
+ # TODO: Check real controller port if config file is specified.
+ status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 9000 for controller ready
+curl localhost:9000/health 1>/dev/null 2>&1
+status=$(echo $?)
+ done
}
function waitForKafkaReady() {
- # TODO: check kafka to be ready instead of sleep
- sleep 10
- echo "kafka is ready"
+ status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 19092 for kafka ready
+echo x |nc localhost 19092 1>/dev/null 2>&1
+status=$(echo $?)
+ done
+}
+
+function waitForBrokerReady() {
+# TODO We are checking for port 8099. Check for real broker port from config
if needed.
+ local status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 8099 for broker ready
+curl localhost:8099/debug/routingTable 1>/dev/null 2>&1
+status=$(echo $?)
+ done
+}
+
+function waitForServerReady() {
+ # TODO We are checjing for port 8097. Check for real server port from config
if needed,
+ local status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 8097 for server ready
+curl localhost:8097/health 1>/dev/null 2>&1
+status=$(echo $?)
+ done
}
function waitForClusterReady() {
- # TODO: check cluster to be ready instead of sleep
- sleep 2
- echo "Cluster ready."
+ waitForBrokerReady
+ waitForServerReady
+ waitForKafkaReady
}
#set config file is present or not
@@ -79,49 +116,68 @@ function setConfigFileArg() {
}
# Given a component and directory, start that version of the specific component
+# Start the service in background.
+# Record the pid file in $2/$1.pid
+# $1 is service name
+# $2 is directory name
+# TODO get rid of exit from this function. Exit only returns from a function.
function startService() {
serviceName=$1
dirName=$2
+ echo Starting $serviceName in $dirName
local configFileArg=$(setConfigFileArg "$3")
# Upon start, save the pid of the process for a component into a file in
/working_dir/{component}.pid, w
[incubator-pinot] branch fix-compat-tests created (now 63de9d9)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch fix-compat-tests in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 63de9d9 Enabling compatibility tests in the script This branch includes the following new commits: new 63de9d9 Enabling compatibility tests in the script The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Enabling compatibility tests in the script
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch fix-compat-tests
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 63de9d99879f6f2a8f575bc879265e9b36d6c48a
Author: Subbu Subramaniam
AuthorDate: Sat May 22 15:59:17 2021 -0700
Enabling compatibility tests in the script
The actual run of compat tests were disahled in the script. This PR
enables the compat tests.
Fixed the verifier script to work with relative as well as absolute
paths for working directory.
Other improvements:
We now preserve the log files of all processes we start.
Changed all pid files to be in one dir so it is easy to stop the cluster
Changed the mechanism to check if some service is up. This part needs more
improvement.
As yet, the tests fail after server upgrade
---
compatibility-verifier/checkoutAndBuild.sh | 4 +-
compatibility-verifier/compCheck.sh| 220 +++--
.../org/apache/pinot/compat/tests/SegmentOp.java | 4 +-
3 files changed, 168 insertions(+), 60 deletions(-)
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
old mode 100644
new mode 100755
index 5d71c21..ec3d7c6
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -37,8 +37,8 @@ function checkoutAndBuild() {
pushd "$targetDir" || exit 1
git init
git remote add origin https://github.com/apache/incubator-pinot
- git fetch --depth 1 origin "$commitHash"
- git checkout FETCH_HEAD
+ git pull origin master
+ git checkout $commitHash
mvn install package -DskipTests -Pbin-dist
popd || exit 1
}
diff --git a/compatibility-verifier/compCheck.sh
b/compatibility-verifier/compCheck.sh
index 6df83a2..0674c0e 100755
--- a/compatibility-verifier/compCheck.sh
+++ b/compatibility-verifier/compCheck.sh
@@ -39,6 +39,8 @@
# one more segment to table, adding some more rows to the stream topic, and
# running some queries with the new data.
+RM="/bin/rm"
+logCount=1
# get usage of the script
function usage() {
@@ -48,27 +50,62 @@ function usage() {
}
function waitForZkReady() {
- # TODO: check ZK to be ready instead of sleep
- sleep 60
- echo "zookeeper is ready"
+ status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 2181 for zk ready
+echo x |nc localhost 2181 1>/dev/null 2>&1
+status=$(echo $?)
+ done
}
function waitForControllerReady() {
- # TODO: check Controller to be ready instead of sleep
- sleep 60
- echo "controller is ready"
+ # TODO: Check real controller port if config file is specified.
+ status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 9000 for controller ready
+curl localhost:9000/health 1>/dev/null 2>&1
+status=$(echo $?)
+ done
}
function waitForKafkaReady() {
- # TODO: check kafka to be ready instead of sleep
- sleep 10
- echo "kafka is ready"
+ status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 19092 for kafka ready
+echo x |nc localhost 19092 1>/dev/null 2>&1
+status=$(echo $?)
+ done
+}
+
+function waitForBrokerReady() {
+# TODO We are checking for port 8099. Check for real broker port from config
if needed.
+ local status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 8099 for broker ready
+curl localhost:8099/debug/routingTable 1>/dev/null 2>&1
+status=$(echo $?)
+ done
+}
+
+function waitForServerReady() {
+ # TODO We are checjing for port 8097. Check for real server port from config
if needed,
+ local status=1
+ while [ $status -ne 0 ]; do
+sleep 1
+echo Checking port 8097 for server ready
+curl localhost:8097/health 1>/dev/null 2>&1
+status=$(echo $?)
+ done
}
function waitForClusterReady() {
- # TODO: check cluster to be ready instead of sleep
- sleep 2
- echo "Cluster ready."
+ waitForBrokerReady
+ waitForServerReady
+ waitForKafkaReady
}
#set config file is present or not
@@ -79,49 +116,68 @@ function setConfigFileArg() {
}
# Given a component and directory, start that version of the specific component
+# Start the service in background.
+# Record the pid file in $2/$1.pid
+# $1 is service name
+# $2 is directory name
+# TODO get rid of exit from this function. Exit only returns from a function.
function startService() {
serviceName=$1
dirName=$2
+ echo Starting $serviceName in $dirName
local configFileArg=$(setConfigFileArg "$3")
# Upon start, save the pid of the process for a component into a file in
/working_dir/{component}.pid, which is then used to stop it
- pushd "$dirName"/pinot-tools/target/pinot-tools-pkg/bin || exit 1
+ pushd "$dirName"/pinot-tools/target/pinot-
[incubator-pinot] branch master updated (1a38329 -> e379292)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from 1a38329 Add complex-type transformation to offline segment creation
(#6914)
add e379292 Add correct yaml files for real time data(#6787) (#6916)
No new revisions were added by this update.
Summary of changes:
...eTest1-schema.json => FeatureTest2-schema.json} | 2 +-
...ata-00.csv => RealtimeFeatureTest1-data-00.csv} | 1 -
... => feature-test-2-realtime-stream-config.json} | 2 +-
...-realtime.json => feature-test-2-realtime.json} | 8 +--
.../queries/feature-test-2-sql-realtime.queries| 62 +
.../feature-test-2-sql-realtime.results| 63 ++
.../sample-test-suite/post-broker-rollback.yaml| 12 +
.../post-controller-rollback.yaml | 12 +
.../sample-test-suite/post-server-rollback.yaml| 12 +
.../sample-test-suite/post-server-upgrade.yaml | 12 +
.../sample-test-suite/pre-broker-upgrade.yaml | 12 +
.../sample-test-suite/pre-controller-upgrade.yaml | 22
.../sample-test-suite/pre-server-upgrade.yaml | 12 +
13 files changed, 226 insertions(+), 6 deletions(-)
copy compatibility-verifier/sample-test-suite/config/{FeatureTest1-schema.json
=> FeatureTest2-schema.json} (98%)
copy
compatibility-verifier/sample-test-suite/config/data/{FeatureTest1-data-00.csv
=> RealtimeFeatureTest1-data-00.csv} (97%)
rename
compatibility-verifier/sample-test-suite/config/{feature-test-1-realtime-stream-config.json
=> feature-test-2-realtime-stream-config.json} (79%)
rename
compatibility-verifier/sample-test-suite/config/{feature-test-1-realtime.json
=> feature-test-2-realtime.json} (87%)
create mode 100644
compatibility-verifier/sample-test-suite/config/queries/feature-test-2-sql-realtime.queries
create mode 100644
compatibility-verifier/sample-test-suite/config/query-results/feature-test-2-sql-realtime.results
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (5faf33d -> 9b44c21)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 5faf33d Adjust threadCpuTime calculation (#6886) add 9b44c21 Add config File support(#6787) (#6901) No new revisions were added by this update. Summary of changes: compatibility-verifier/compCheck.sh| 40 ++- .../sample-test-suite/config/kafka/topic1.cfg | 24 - .../sample-test-suite/kafka-stream-produce-op.yaml | 29 --- .../sample-test-suite/kafka-topic-create-op.yaml | 25 -- .../sample-test-suite/offline-table-create-op.yaml | 26 -- .../sample-test-suite/offline-table-delete-op.yaml | 26 -- .../sample-test-suite/post-broker-rollback.yaml| 41 +--- .../post-controller-rollback.yaml | 51 +-- .../sample-test-suite/post-server-rollback.yaml| 41 +--- .../sample-test-suite/post-server-upgrade.yaml | 41 +--- .../sample-test-suite/pre-broker-upgrade.yaml | 41 +--- .../sample-test-suite/pre-controller-upgrade.yaml | 46 +++-- .../sample-test-suite/pre-server-upgrade.yaml | 41 +--- .../sample-test-suite/query-op.yaml| 25 -- .../realtime-table-create-op.yaml | 26 -- .../realtime-table-delete-op.yaml | 26 -- .../sample-test-suite/sample.yaml | 57 -- .../sample-test-suite/segment-create-op.yaml | 29 --- .../sample-test-suite/segment-delete-op.yaml | 26 -- .../org/apache/pinot/compat/tests/QueryOp.java | 7 ++- 20 files changed, 139 insertions(+), 529 deletions(-) delete mode 100644 compatibility-verifier/sample-test-suite/config/kafka/topic1.cfg delete mode 100644 compatibility-verifier/sample-test-suite/kafka-stream-produce-op.yaml delete mode 100644 compatibility-verifier/sample-test-suite/kafka-topic-create-op.yaml delete mode 100644 compatibility-verifier/sample-test-suite/offline-table-create-op.yaml delete mode 100644 compatibility-verifier/sample-test-suite/offline-table-delete-op.yaml delete mode 100644 compatibility-verifier/sample-test-suite/query-op.yaml delete mode 100644 compatibility-verifier/sample-test-suite/realtime-table-create-op.yaml delete mode 100644 compatibility-verifier/sample-test-suite/realtime-table-delete-op.yaml delete mode 100644 compatibility-verifier/sample-test-suite/sample.yaml delete mode 100644 compatibility-verifier/sample-test-suite/segment-create-op.yaml delete mode 100644 compatibility-verifier/sample-test-suite/segment-delete-op.yaml - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Add absolute path support for compatibility testing(#6787) (#6828)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new b4d75a3 Add absolute path support for compatibility testing(#6787)
(#6828)
b4d75a3 is described below
commit b4d75a30ccc26b30b0779433c5f009afac5b5588
Author: Sharayu
AuthorDate: Mon Apr 26 20:24:28 2021 -0700
Add absolute path support for compatibility testing(#6787) (#6828)
---
compatibility-verifier/compCheck.sh| 32
.../config}/FeatureTest1-schema.json | 0
.../config}/data/FeatureTest1-data-00.csv | 0
.../config}/data/FeatureTest1-data-01.csv | 0
.../config}/data/recordReaderConfig.json | 0
.../feature-test-1-realtime-stream-config.json | 0
.../config}/feature-test-1-realtime.json | 0
.../sample-test-suite/config}/feature-test-1.json | 4 +-
.../sample-test-suite/config}/kafka/topic1.cfg | 0
.../config}/queries/feature-test-1-sql.queries | 0
.../query-results/feature-test-1-rest-sql.results | 0
.../kafka-stream-produce-op.yaml | 8 +--
.../sample-test-suite}/kafka-topic-create-op.yaml | 2 +-
.../offline-table-create-op.yaml | 4 +-
.../offline-table-delete-op.yaml | 4 +-
.../sample-test-suite}/post-broker-rollback.yaml | 0
.../post-controller-rollback.yaml | 0
.../sample-test-suite}/post-server-rollback.yaml | 0
.../sample-test-suite}/post-server-upgrade.yaml| 0
.../sample-test-suite}/pre-broker-upgrade.yaml | 0
.../sample-test-suite}/pre-controller-upgrade.yaml | 0
.../sample-test-suite}/pre-server-upgrade.yaml | 0
.../sample-test-suite}/query-op.yaml | 4 +-
.../realtime-table-create-op.yaml | 4 +-
.../realtime-table-delete-op.yaml | 4 +-
.../sample-test-suite/sample.yaml | 30 +++-
.../sample-test-suite}/segment-create-op.yaml | 8 +--
.../sample-test-suite}/segment-delete-op.yaml | 4 +-
pinot-integration-tests/pom.xml| 4 ++
.../java/org/apache/pinot/compat/tests/BaseOp.java | 14 ++
.../pinot/compat/tests/CompatibilityOpsRunner.java | 16 --
.../org/apache/pinot/compat/tests/QueryOp.java | 4 +-
.../org/apache/pinot/compat/tests/SegmentOp.java | 10 ++--
.../org/apache/pinot/compat/tests/StreamOp.java| 10 ++--
.../org/apache/pinot/compat/tests/TableOp.java | 6 +--
.../src/test/resources/compat-tests/sample.yaml| 57 --
36 files changed, 109 insertions(+), 120 deletions(-)
diff --git a/compatibility-verifier/compCheck.sh
b/compatibility-verifier/compCheck.sh
index 9660ec9..ad168f5 100755
--- a/compatibility-verifier/compCheck.sh
+++ b/compatibility-verifier/compCheck.sh
@@ -43,7 +43,7 @@
# get usage of the script
function usage() {
command=$1
- echo "Usage: $command [workingDir]"
+ echo "Usage: $command workingDir testSuiteDir"
exit 1
}
@@ -146,11 +146,21 @@ function setupCompatTester() {
export CLASSPATH_PREFIX
}
+#compute absolute path for testSuiteDir if given relative
+function absPath() {
+ local testSuiteDirPath=$1
+ if [[ ! "$testSuiteDirPath" = /* ]]; then
+#relative path
+testSuiteDirPath=$(cd "$testSuiteDirPath"; pwd)
+ fi
+ echo "$testSuiteDirPath"
+}
+
#
# Main
#
-if [ $# -ne 1 ] ; then
+if [ $# -ne 2 ] ; then
usage compCheck
fi
@@ -158,6 +168,7 @@
COMPAT_TESTER_PATH="pinot-integration-tests/target/pinot-integration-tests-pkg/b
# create subdirectories for given commits
workingDir=$1
+testSuiteDir=$(absPath "$2")
oldTargetDir="$workingDir"/oldTargetDir
newTargetDir="$workingDir"/newTargetDir
@@ -172,30 +183,31 @@ fi
# Setup initial cluster with olderCommit and do rolling upgrade
+# Provide abspath of filepath to $COMPAT_TESTER
startServices "$oldTargetDir"
-#$COMPAT_TESTER pre-controller-upgrade.yaml 1; if [ $? -ne 0 ]; then exit 1; fi
+#$COMPAT_TESTER $testSuiteDir/pre-controller-upgrade.yaml 1; if [ $? -ne 0 ];
then exit 1; fi
stopService controller "$oldTargetDir"
startService controller "$newTargetDir"
waitForControllerReady
-#$COMPAT_TESTER pre-broker-upgrade.yaml 2; if [ $? -ne 0 ]; then exit 1; fi
+#$COMPAT_TESTER $testSuiteDir/pre-broker-upgrade.yaml 2; if [ $? -ne 0 ]; then
exit 1; fi
stopService broker "$oldTargetDir"
startService broker "$newTargetDir"
-#$COMPAT_TESTER pre-server-upgrade.yaml 3; if [ $? -ne 0 ]; then exit 1; fi
+#$COMPAT_TESTER $testSuiteDir/pre-server-upgrade.yaml 3; if [ $? -ne 0 ]; then
exit 1; fi
stopService server "$oldTargetDir"
startService server "$newTargetDir"
-#$COMPAT_TESTE
[incubator-pinot] branch master updated: Fix log message for SegmentBuildTimeLeaseExtender (#6684)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 621ceb0 Fix log message for SegmentBuildTimeLeaseExtender (#6684)
621ceb0 is described below
commit 621ceb021cefa1640708f3bb4d576270717f70d6
Author: Jiapeng Tao
AuthorDate: Wed Mar 17 09:42:15 2021 -0700
Fix log message for SegmentBuildTimeLeaseExtender (#6684)
---
.../core/data/manager/realtime/SegmentBuildTimeLeaseExtender.java | 8 ++--
.../pinot/server/starter/helix/HelixInstanceDataManager.java | 2 --
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentBuildTimeLeaseExtender.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentBuildTimeLeaseExtender.java
index 725dc95..6dc2008 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentBuildTimeLeaseExtender.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentBuildTimeLeaseExtender.java
@@ -56,6 +56,7 @@ public class SegmentBuildTimeLeaseExtender {
public static void initExecutor() {
_executor = new ScheduledThreadPoolExecutor(1);
+LOGGER.info("Initialized segment build time lease extender executor");
}
public static void shutdownExecutor() {
@@ -63,6 +64,7 @@ public class SegmentBuildTimeLeaseExtender {
_executor.shutdownNow();
_executor = null;
}
+LOGGER.info("Shut down segment build time lease extender executor");
}
@VisibleForTesting
@@ -78,9 +80,11 @@ public class SegmentBuildTimeLeaseExtender {
ServerMetrics serverMetrics, String tableNameWithType) {
return TABLE_TO_LEASE_EXTENDER.compute(tableNameWithType, (k, v) -> {
if (v == null) {
-return new SegmentBuildTimeLeaseExtender(instanceId, serverMetrics,
tableNameWithType);
+SegmentBuildTimeLeaseExtender leaseExtender = new
SegmentBuildTimeLeaseExtender(instanceId, serverMetrics, tableNameWithType);
+LOGGER.info("Created lease extender for table: {}", tableNameWithType);
+return leaseExtender;
} else {
-LOGGER.warn("Lease extender for Table: {} already exists",
tableNameWithType);
+LOGGER.warn("Lease extender for table: {} already exists",
tableNameWithType);
return v;
}
});
diff --git
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
index 30ba125..c0dc9df 100644
---
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
+++
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
@@ -94,7 +94,6 @@ public class HelixInstanceDataManager implements
InstanceDataManager {
// Initialize segment build time lease extender executor
SegmentBuildTimeLeaseExtender.initExecutor();
-LOGGER.info("Initialized segment build time lease extender executor");
// Initialize the table data manager provider
TableDataManagerProvider.init(_instanceDataManagerConfig);
@@ -113,7 +112,6 @@ public class HelixInstanceDataManager implements
InstanceDataManager {
tableDataManager.shutDown();
}
SegmentBuildTimeLeaseExtender.shutdownExecutor();
-LOGGER.info("Segment build time lease extender executor shut down");
LOGGER.info("Helix instance data manager shut down");
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: complete compatibility regression testing (#6650)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 59d02c6 complete compatibility regression testing (#6650)
59d02c6 is described below
commit 59d02c66af3ab828bb3f75c0f832d03db0917f1a
Author: Liang Mingqiang
AuthorDate: Wed Mar 10 13:40:03 2021 -0800
complete compatibility regression testing (#6650)
* split compCheck.sh as two script so we can build and test separately
* redirect services' logs to a file for easy of debug
* wait for zk/controller to be ready
* fix some minor bugs.
---
compatibility-verifier/README.md | 38 ++
compatibility-verifier/checkoutAndBuild.sh | 79 +++
compatibility-verifier/compCheck.sh| 150 -
.../org/apache/pinot/compat/tests/SegmentOp.java | 8 +-
.../pinot/compat/tests/SqlResultComparator.java| 35 +++--
.../compat-tests/configs/feature-test-1.json | 4 +-
.../compat-tests/data/FeatureTest1-data-00.csv | 25 ++--
.../compat-tests/data/FeatureTest1-data-01.csv | 17 ++-
.../compat-tests/data/recordReaderConfig.json | 2 +-
...p-publish.yaml => kafka-stream-produce-op.yaml} | 12 +-
.../compat-tests/kafka-topic-create-op.yaml| 25
...p-publish.yaml => offline-table-create-op.yaml} | 19 +--
...p-publish.yaml => offline-table-delete-op.yaml} | 21 +--
.../queries/feature-test-1-sql.queries | 17 ++-
.../src/test/resources/compat-tests/query-op.yaml | 28 +---
.../query-results/feature-test-1-rest-sql.results | 1 -
...-publish.yaml => realtime-table-create-op.yaml} | 17 +--
...-publish.yaml => realtime-table-delete-op.yaml} | 21 +--
...ream-op-publish.yaml => segment-create-op.yaml} | 24 +---
.../resources/compat-tests/segment-delete-op.yaml | 26
.../resources/compat-tests/table-segment-op.yaml | 45 ---
21 files changed, 302 insertions(+), 312 deletions(-)
diff --git a/compatibility-verifier/README.md b/compatibility-verifier/README.md
new file mode 100644
index 000..0f6c9b7
--- /dev/null
+++ b/compatibility-verifier/README.md
@@ -0,0 +1,38 @@
+
+
+# Compatibility Regression Testing Scripts for Apache Pinot
+
+## Usage
+
+### Step 1: checkout source code and build targets for older commit and newer
commit
+```shell
+./compatibility-verifier/checkoutAndBuild.sh [olderCommit] [newerCommit]
[workingDir]
+```
+***NOTE***: `[workingDir]` is optional, if user does not specify
`[workingDir]`, the script will create a temporary working
+dir and output the path, which can be used in step 2.
+
+### Step 2: run compatibility regression test against the two targets build in
step1
+```shell
+./compatibility-verifier/compCheck.sh [workingDir]
+```
+***NOTE***: the script can only be run under the root folder of the project
currently. Before run the script, make sure to
+change to the right directory first.
diff --git a/compatibility-verifier/checkoutAndBuild.sh
b/compatibility-verifier/checkoutAndBuild.sh
new file mode 100644
index 000..5d71c21
--- /dev/null
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -0,0 +1,79 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+
+# get a temporary directory in case the workingDir is not provided by user
+TMP_DIR=$(mktemp -d 2>/dev/null || mktemp -d -t 'mytmpdir')
+
+# get usage of the script
+function usage() {
+ command=$1
+ echo "Usage: $command olderCommit newerCommit [workingDir]"
+ exit 1
+}
+
+# This function builds Pinot given a specific commit hash and target directory
+function checkoutAndBuild() {
+ commitHash=$1
+ targetDir=$2
+
+ pushd "$targetDir" || exit 1
+ git init
+ git remote add origin https://github.com/apache/incubator-pinot
+ git fetch --depth 1 origin "$commitHash"
+ git checkout FETCH_HEAD
+ mvn install package -DskipTests -Pbin-dist
+ popd || exit 1
+}
+
+# get arguments
+olderCommit=$1
+newerCommit=$2
+
+if [ -n "$3" ]; then
+ workingDir=
[incubator-pinot] branch master updated (6644968 -> a73c523)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 6644968 Support identifier/literal as the derived column transform function (#6657) add a73c523 remove duplicate code (#6643) No new revisions were added by this update. Summary of changes: .../java/org/apache/pinot/compat/tests/BaseOp.java | 1 + .../org/apache/pinot/compat/tests/QueryOp.java | 3 +- .../apache/pinot/compat/tests/QueryProcessor.java | 98 -- .../org/apache/pinot/compat/tests/SegmentOp.java | 8 +- .../org/apache/pinot/compat/tests/StreamOp.java| 4 +- .../pinot/integration/tests/ClusterTest.java | 2 +- 6 files changed, 9 insertions(+), 107 deletions(-) delete mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/QueryProcessor.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (df767c0 -> b2d716d)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from df767c0 Removing redundant check for column with RAW encoding and
noDictionary config (#6636)
add b2d716d Add generation number to data and queries (#6638)
No new revisions were added by this update.
Summary of changes:
compatibility-verifier/compCheck.sh| 29 ++---
.../java/org/apache/pinot/compat/tests/BaseOp.java | 6 +-
.../pinot/compat/tests/CompatibilityOpsRunner.java | 12 +-
.../org/apache/pinot/compat/tests/QueryOp.java | 10 +-
.../org/apache/pinot/compat/tests/SegmentOp.java | 26 -
.../org/apache/pinot/compat/tests/StreamOp.java| 44 ---
.../org/apache/pinot/compat/tests/TableOp.java | 2 +-
.../tests/{CompatTestOperation.java => Utils.java} | 40 ---
.../compat-tests/configs/FeatureTest1-schema.json | 13 ++-
.../compat-tests/configs/feature-test-1.json | 1 +
.../compat-tests/data/FeatureTest1-data-00.csv | 22 ++--
.../compat-tests/data/FeatureTest1-data-01.csv | 16 +--
.../compat-tests/data/recordReaderConfig.json | 2 +-
.../queries/feature-test-1-sql.queries | 128 ++---
.../query-results/feature-test-1-rest-sql.results | 2 +-
.../resources/compat-tests/stream-op-publish.yaml | 26 +++--
16 files changed, 215 insertions(+), 164 deletions(-)
copy
pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/{CompatTestOperation.java
=> Utils.java} (51%)
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (a7c4259 -> df767c0)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from a7c4259 LICENSE and NOTICE changes for 0.7 release (#6623) add df767c0 Removing redundant check for column with RAW encoding and noDictionary config (#6636) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/pinot/core/util/TableConfigUtils.java| 4 1 file changed, 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Implement QueryOp class. (#6583)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 7a989f7 Implement QueryOp class. (#6583)
7a989f7 is described below
commit 7a989f72d495c4c72904fa3b811259ac6e37043d
Author: Liang Mingqiang
AuthorDate: Fri Feb 26 11:21:30 2021 -0800
Implement QueryOp class. (#6583)
* Implement QueryOp class.
Implement QueryOp class, which take a file that has queries, and
a file that has results, run the queries through the broker/server
and compare the results with the result file.
* address comments: reformat code; verify routing table is updated
in SegmentOp
* address jackjlli's comments
* address mcvsubbu's comments
* address jackjlli's comments: throwing a generic Exception
* use println instead of printf
Co-authored-by: Ming Liang
---
.../pinot/compat/tests/CompatibilityOpsRunner.java | 11 +-
.../org/apache/pinot/compat/tests/QueryOp.java | 95 +-
.../apache/pinot/compat/tests/QueryProcessor.java | 98 ++
.../org/apache/pinot/compat/tests/SegmentOp.java | 26 +-
.../pinot/compat/tests/SqlResultComparator.java| 353 +
.../compat-tests/configs/feature-test-1.json | 11 +
.../queries/feature-test-1-sql.queries | 89 ++
.../src/test/resources/compat-tests/query-op.yaml | 49 +++
.../query-results/feature-test-1-rest-sql.results | 85 +
.../resources/compat-tests/table-segment-op.yaml | 5 +
10 files changed, 812 insertions(+), 10 deletions(-)
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/CompatibilityOpsRunner.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/CompatibilityOpsRunner.java
index fe9144c..7350cf9 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/CompatibilityOpsRunner.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/CompatibilityOpsRunner.java
@@ -18,14 +18,10 @@
*/
package org.apache.pinot.compat.tests;
-import com.fasterxml.jackson.core.JsonParseException;
-import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
-import java.io.IOException;
import java.io.InputStream;
-import org.apache.pinot.compat.tests.BaseOp;
-import org.apache.pinot.compat.tests.CompatTestOperation;
+
public class CompatibilityOpsRunner {
private static final String ROOT_DIR = "compat-tests";
@@ -36,8 +32,7 @@ public class CompatibilityOpsRunner {
_configFileName = configFileName;
}
- private boolean runOps()
- throws IOException, JsonParseException, JsonMappingException {
+ private boolean runOps() throws Exception {
String filePath = ROOT_DIR + "/" + _configFileName;
InputStream inputStream =
getClass().getClassLoader().getResourceAsStream(filePath);
@@ -56,7 +51,7 @@ public class CompatibilityOpsRunner {
return passed;
}
- public static void main(String[] args) throws Exception {
+ public static void main(String[] args) throws Exception {
if (args.length < 1 || args.length > 1) {
throw new IllegalArgumentException("Need exactly one file name as
argument");
}
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/QueryOp.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/QueryOp.java
index 48e4ba0..eeafb15 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/QueryOp.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/QueryOp.java
@@ -19,6 +19,14 @@
package org.apache.pinot.compat.tests;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.BufferedReader;
+import java.io.FileInputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
@@ -30,6 +38,11 @@ import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
*/
@JsonIgnoreProperties(ignoreUnknown = true)
public class QueryOp extends BaseOp {
+ private static final Logger LOGGER = LoggerFactory.getLogger(QueryOp.class);
+
+ private static final String NUM_DOCS_SCANNED_KEY = "numDocsScanned";
+ private static final String TIME_USED_MS_KEY = "timeUsedMs";
+ private static final String COMMENT_DELIMITER = "#";
private String _queryFileName;
private String _expectedResultsFileName;
@@ -37,6 +50,11 @@ public class QueryOp extends BaseOp {
super(OpType.QUERY_O
[incubator-pinot] branch master updated (e36159d -> 2b3bf29)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from e36159d Adding a controller periodic task to clean up dead minion instances (#6543) add 2b3bf29 Introduce a metric for query/response size on broker. (#6590) No new revisions were added by this update. Summary of changes: .../apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java | 3 +++ .../src/main/java/org/apache/pinot/common/metrics/BrokerGauge.java | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (d0177f9 -> abf21ed)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from d0177f9 [TE]: Fix all eslint reported errors and warnings under app/pods/components (#6568) add abf21ed Compatibility test: streamOp (#6419) No new revisions were added by this update. Summary of changes: compatibility-verifier/compCheck.sh| 14 +- .../pinot/compat/tests/ClusterDescriptor.java | 17 +- .../org/apache/pinot/compat/tests/StreamOp.java| 231 +++-- .../compat-tests/configs/FeatureTest1-schema.json | 7 + .../feature-test-1-realtime-stream-config.json | 8 + .../configs/feature-test-1-realtime.json | 59 ++ .../resources/compat-tests/stream-op-publish.yaml | 7 +- 7 files changed, 315 insertions(+), 28 deletions(-) create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/configs/feature-test-1-realtime-stream-config.json create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/configs/feature-test-1-realtime.json - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (43a3701 -> 27e34d7)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 43a3701 [TE]frontend - Change the formatting of metric display in exploration page (#6545) add 27e34d7 Add Access Control for REST endpoints of Controller - Declarative Approach (#6507) No new revisions were added by this update. Summary of changes: .../api/ControllerAdminApiApplication.java | 2 + .../pinot/controller/api/access/AccessControl.java | 31 +- .../controller/api/access/AccessControlUtils.java | 98 ++ .../pinot/controller/api/access/AccessType.java| 13 +-- .../pinot/controller/api/access/Authenticate.java | 18 ++-- .../api/access/AuthenticationFilter.java | 100 +++ .../resources/LLCSegmentCompletionHandlers.java| 5 + .../api/resources/PinotBrokerRestletResource.java | 3 + .../api/resources/PinotClusterConfigs.java | 4 + .../resources/PinotIngestionRestletResource.java | 4 + .../PinotInstanceAssignmentRestletResource.java| 6 ++ .../resources/PinotInstanceRestletResource.java| 7 ++ .../api/resources/PinotSchemaRestletResource.java | 27 - .../api/resources/PinotSegmentRestletResource.java | 13 +++ .../PinotSegmentUploadDownloadRestletResource.java | 8 ++ .../api/resources/PinotTableIndexingConfigs.java | 3 + .../api/resources/PinotTableMetadataConfigs.java | 3 + .../api/resources/PinotTableRestletResource.java | 26 - .../api/resources/PinotTableSegmentConfigs.java| 3 + .../api/resources/PinotTableTenantConfigs.java | 3 + .../api/resources/PinotTaskRestletResource.java| 11 +++ .../api/resources/PinotTenantRestletResource.java | 6 ++ .../api/resources/ZookeeperResource.java | 4 + .../api/access/AuthenticationFilterTest.java | 109 + 24 files changed, 483 insertions(+), 24 deletions(-) create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessControlUtils.java copy pinot-tools/src/main/java/org/apache/pinot/tools/data/generator/Generator.java => pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AccessType.java (82%) copy pinot-spi/src/main/java/org/apache/pinot/spi/annotations/minion/TaskExecutorFactory.java => pinot-controller/src/main/java/org/apache/pinot/controller/api/access/Authenticate.java (70%) create mode 100644 pinot-controller/src/main/java/org/apache/pinot/controller/api/access/AuthenticationFilter.java create mode 100644 pinot-controller/src/test/java/org/apache/pinot/controller/api/access/AuthenticationFilterTest.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (9708292 -> c736fbe)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from 9708292 Move SegmentGenerationUtils from module
pinot-ingestion-common to pinot-common (#6537)
add c736fbe Changed the compat script to run single yaml file (#6550)
No new revisions were added by this update.
Summary of changes:
compatibility-verifier/compCheck.sh| 24 ++
.../test/resources/compat-tests/kafka/topic1.cfg | 5
.../resources/compat-tests/stream-op-publish.yaml | 29 +-
.../{sample.yaml => table-segment-op.yaml} | 25 +++
4 files changed, 45 insertions(+), 38 deletions(-)
create mode 100644
pinot-integration-tests/src/test/resources/compat-tests/kafka/topic1.cfg
copy .github/workflows/pinot_tests-workflow-run.yml =>
pinot-integration-tests/src/test/resources/compat-tests/stream-op-publish.yaml
(58%)
copy pinot-integration-tests/src/test/resources/compat-tests/{sample.yaml =>
table-segment-op.yaml} (72%)
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Guard against multiple consuming segments for same partition (#6483)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new bacaed2 Guard against multiple consuming segments for same partition
(#6483)
bacaed2 is described below
commit bacaed22b4aa3eef38f951fdb098396706df444e
Author: Subbu Subramaniam
AuthorDate: Tue Jan 26 09:00:54 2021 -0800
Guard against multiple consuming segments for same partition (#6483)
* Guard against multiple consuming segments for same partition
Fixes being considered for issues #5559 and #5263 may end up
resurfacing some race conditions that we have strived to avoid
via automated mechanisms. The race conditions may lead to multiple
consuming segments in the same partition.
This commit is a catch-all safeguard in case some automaticl operation
interferes with other manual primitives provided, so that the table
never reaches such a state.
* Addressed review comments
---
.../core/realtime/PinotLLCRealtimeSegmentManager.java | 19 +++
1 file changed, 19 insertions(+)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
index 00433b3..42d50d0 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
@@ -793,6 +793,25 @@ public class PinotLLCRealtimeSegmentManager {
LOGGER.info("Updating segment: {} to ONLINE state",
committingSegmentName);
}
+// There used to be a race condition in pinot (caused by heavy GC on the
controller during segment commit)
+// that ended up creating multiple consuming segments for the same stream
partition, named somewhat like
+// tableName__1__25__20210920T190005Z and
tableName__1__25__20210920T190007Z. It was fixed by checking the
+// Zookeeper Stat object before updating the segment metadata.
+// These conditions can happen again due to manual operations considered
as fixes in Issues #5559 and #5263
+// The following check prevents the table from going into such a state
(but does not prevent the root cause
+// of attempting such a zk update).
+LLCSegmentName newLLCSegmentName = new LLCSegmentName(newSegmentName);
+int partitionId = newLLCSegmentName.getPartitionId();
+int seqNum = newLLCSegmentName.getSequenceNumber();
+for (String segmentNameStr : instanceStatesMap.keySet()) {
+ LLCSegmentName llcSegmentName = new LLCSegmentName(segmentNameStr);
+ if (llcSegmentName.getPartitionId() == partitionId &&
llcSegmentName.getSequenceNumber() == seqNum) {
+String errorMsg =
+String.format("Segment %s is a duplicate of existing segment %s",
newSegmentName, segmentNameStr);
+LOGGER.error(errorMsg);
+throw new HelixHelper.PermanentUpdaterException(errorMsg);
+ }
+}
// Assign instances to the new segment and add instances as state CONSUMING
List instancesAssigned =
segmentAssignment.assignSegment(newSegmentName, instanceStatesMap,
instancePartitionsMap);
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (830eb94 -> ec15f41)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 830eb94 [TE]frontend - Activate the composite anomalies route (#6472) add ec15f41 Bump up quartz version to 2.3.2 to prevent XXE (#6484) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (02ab193 -> 8fcb17d)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 02ab193 remove accidental junit imports (#6404) add 8fcb17d Compatibility test for segment operations upload and delete (#6382) No new revisions were added by this update. Summary of changes: .../helix/ControllerRequestURLBuilder.java | 13 +- .../pinot/compat/tests/ClusterDescriptor.java | 10 +- .../org/apache/pinot/compat/tests/SegmentOp.java | 256 +++-- .../org/apache/pinot/compat/tests/TableOp.java | 26 ++- .../compat-tests/configs/feature-test-1.json | 28 +-- .../src/test/resources/compat-tests/sample.yaml| 16 +- 6 files changed, 290 insertions(+), 59 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (7580ae4 -> 33de6dc)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 7580ae4 [TE]frontend - Integrate Performance Stats into Composite Anomalies (#6366) add 33de6dc Introduce DimensionTableDataManager (#6346) No new revisions were added by this update. Summary of changes: .../core/data/manager/SegmentDataManager.java | 2 +- .../manager/offline/DimensionTableDataManager.java | 164 +++ .../manager/offline/TableDataManagerProvider.java | 7 +- .../apache/pinot/core/util/TableConfigUtils.java | 20 ++- .../offline/DimensionTableDataManagerTest.java | 173 + .../pinot/core/util/TableConfigUtilsTest.java | 88 +++ .../src/test/resources/data/dimBaseballTeams.avro | Bin 0 -> 2214 bytes .../src/test/resources/data/dimBaseballTeams.avsc | 8 + .../src/test/resources/data/dimBaseballTeams.csv | 52 +++ .../resources/data/dimBaseballTeams_schema.json| 14 ++ .../org/apache/pinot/tools/JoinQuickStart.java | 124 +++ 11 files changed, 649 insertions(+), 3 deletions(-) create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManager.java create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/data/manager/offline/DimensionTableDataManagerTest.java create mode 100644 pinot-core/src/test/resources/data/dimBaseballTeams.avro create mode 100644 pinot-core/src/test/resources/data/dimBaseballTeams.avsc create mode 100644 pinot-core/src/test/resources/data/dimBaseballTeams.csv create mode 100644 pinot-core/src/test/resources/data/dimBaseballTeams_schema.json create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/JoinQuickStart.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: compatibility test: create/delete table operations (#6360)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new b8bc74f compatibility test: create/delete table operations (#6360)
b8bc74f is described below
commit b8bc74fa79437a4f30cf1daed55f66b48973b7f9
Author: Jiapeng Tao <50065407+jta...@users.noreply.github.com>
AuthorDate: Mon Dec 21 19:57:27 2020 -0800
compatibility test: create/delete table operations (#6360)
Co-authored-by: Jiapeng Tao
---
.../pinot/compat/tests/ClusterDescriptor.java | 23 +
.../org/apache/pinot/compat/tests/TableOp.java | 55 --
2 files changed, 73 insertions(+), 5 deletions(-)
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/ClusterDescriptor.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/ClusterDescriptor.java
new file mode 100644
index 000..46dbcd0
--- /dev/null
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/ClusterDescriptor.java
@@ -0,0 +1,23 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.compat.tests;
+
+public class ClusterDescriptor {
+ public static final String CONTROLLER_URL = "localhost:9000";
+}
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/TableOp.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/TableOp.java
index dfb9ab1..1eeb01e 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/TableOp.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/TableOp.java
@@ -19,6 +19,15 @@
package org.apache.pinot.compat.tests;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import java.io.File;
+import java.io.IOException;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.controller.helix.ControllerRequestURLBuilder;
+import org.apache.pinot.controller.helix.ControllerTest;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
@@ -45,6 +54,8 @@ public class TableOp extends BaseOp {
private Op _op;
private String _tableConfigFileName;
+ private static final Logger LOGGER = LoggerFactory.getLogger(TableOp.class);
+
public TableOp() {
super(OpType.TABLE_OP);
}
@@ -77,12 +88,10 @@ public class TableOp extends BaseOp {
boolean runOp() {
switch (_op) {
case CREATE:
-System.out.println("Creating table from table config " +
_tableConfigFileName + " and schema " +
-_schemaFileName);
-break;
+createSchema();
+return createTable();
case DELETE:
-System.out.println("Deleting table that has table config " +
_tableConfigFileName);
-break;
+return deleteTable();
case UPDATE_CONFIG:
System.out.println("Updating table config to " + _tableConfigFileName);
break;
@@ -92,4 +101,40 @@ public class TableOp extends BaseOp {
}
return true;
}
+
+ private boolean createSchema() {
+try {
+ ControllerTest.sendPostRequest(
+
ControllerRequestURLBuilder.baseUrl(ClusterDescriptor.CONTROLLER_URL).forSchemaCreate(),
+ FileUtils.readFileToString(new File(_schemaFileName)));
+ return true;
+} catch (IOException e) {
+ LOGGER.error("Failed to create schema with file: {}", _schemaFileName,
e);
+ return false;
+}
+ }
+
+ private boolean createTable() {
+try {
+ ControllerTest.sendPostRequest(
+
ControllerRequestURLBuilder.baseUrl(ClusterDescriptor.CONTROLLER_URL).forTableCreate(),
+ FileUtils.readFileToString(new File(_tableConfigFileName)));
+ return true;
+} catch (IOException e) {
+ LOGGER.error("Failed to create table with file: {}",
_tableConfigFileName, e);
+ return false;
+}
+ }
+
+ private bool
[incubator-pinot] branch master updated (64d1054 -> 75f9fd3)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from 64d1054 Adding offline dim table creation and assignment (#6286)
add 75f9fd3 Add a Controller endpoint to return table creation time
(#6331)
No new revisions were added by this update.
Summary of changes:
.../api/resources/PinotTableRestletResource.java | 55 +++---
.../helix/core/PinotHelixResourceManager.java | 13 +
.../{TableCustomConfig.java => TableStats.java}| 28 ++-
3 files changed, 67 insertions(+), 29 deletions(-)
copy
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/{TableCustomConfig.java
=> TableStats.java} (64%)
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Setup classpath correctly for the compat test runner (#6160)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new cf675a5 Setup classpath correctly for the compat test runner (#6160)
cf675a5 is described below
commit cf675a5f9e09df12dd2726786d91cb214046f931
Author: Subbu Subramaniam
AuthorDate: Tue Oct 20 08:49:10 2020 -0700
Setup classpath correctly for the compat test runner (#6160)
---
compatibility-verifier/compCheck.sh | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/compatibility-verifier/compCheck.sh
b/compatibility-verifier/compCheck.sh
index 238a8d1..16465ef 100755
--- a/compatibility-verifier/compCheck.sh
+++ b/compatibility-verifier/compCheck.sh
@@ -125,6 +125,15 @@ function stopServices() {
echo "Cluster stopped."
}
+# Setup the path and classpath prefix for compatibility tester executable
+function setupCompatTester() {
+ COMPAT_TESTER="$(dirname $0)/../${COMPAT_TESTER_PATH}"
+ local pinotIntegTestsRelDir="$(dirname $0)/../pinot-integration-tests/target"
+ local pinotIntegTestsAbsDir=`(cd ${pinotIntegTestsRelDir};pwd)`
+ CLASSPATH_PREFIX=$(ls
${pinotIntegTestsAbsDir}/pinot-integration-tests-*-tests.jar)
+ export CLASSPATH_PREFIX
+}
+
#
# Main
#
@@ -132,7 +141,7 @@ function stopServices() {
# cleanp the temporary directory when the bash script exits
trap cleanup EXIT
-COMPAT_TESTER="$(dirname $0)/../${COMPAT_TESTER_PATH}"
+setupCompatTester
if [ $# -lt 2 ] || [ $# -gt 3 ] ; then
usage compCheck
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (b22045a -> e15fa80)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from b22045a Do not apply metadata/dictionary based agg operator to upsert table (#6154) add e15fa80 Framework for adding compatibility tests (#6129) No new revisions were added by this update. Summary of changes: compatibility-verifier/compCheck.sh| 36 +++- pinot-integration-tests/pom.xml| 8 ++ .../java/org/apache/pinot/compat/tests/BaseOp.java | 73 + .../pinot/compat/tests/CompatTestOperation.java| 25 +++--- .../pinot/compat/tests/CompatibilityOpsRunner.java | 71 .../org/apache/pinot/compat/tests/QueryOp.java | 61 ++ .../org/apache/pinot/compat/tests/SegmentOp.java | 95 ++ .../org/apache/pinot/compat/tests/StreamOp.java| 88 .../org/apache/pinot/compat/tests/TableOp.java | 95 ++ .../compat-tests/configs/FeatureTest1-schema.json | 82 +++ .../compat-tests/configs/feature-test-1.json | 54 .../compat-tests/data/FeatureTest1-data-00.csv | 13 +++ .../compat-tests/data/FeatureTest1-data-01.csv | 10 +++ .../compat-tests/data/recordReaderConfig.json | 5 ++ .../compat-tests/post-broker-rollback.yaml | 49 +++ .../compat-tests/post-controller-rollback.yaml | 49 +++ .../compat-tests/post-server-rollback.yaml | 49 +++ .../compat-tests/post-server-upgrade.yaml | 49 +++ .../resources/compat-tests/pre-broker-upgrade.yaml | 49 +++ .../compat-tests/pre-controller-upgrade.yaml | 49 +++ .../resources/compat-tests/pre-server-upgrade.yaml | 49 +++ .../src/test/resources/compat-tests/sample.yaml| 49 +++ 22 files changed, 1095 insertions(+), 13 deletions(-) create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/BaseOp.java copy pinot-common/src/main/java/org/apache/pinot/common/restlet/resources/TableSizeInfo.java => pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/CompatTestOperation.java (68%) create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/CompatibilityOpsRunner.java create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/QueryOp.java create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/SegmentOp.java create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/StreamOp.java create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/compat/tests/TableOp.java create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/configs/FeatureTest1-schema.json create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/configs/feature-test-1.json create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/data/FeatureTest1-data-00.csv create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/data/FeatureTest1-data-01.csv create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/data/recordReaderConfig.json create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/post-broker-rollback.yaml create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/post-controller-rollback.yaml create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/post-server-rollback.yaml create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/post-server-upgrade.yaml create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/pre-broker-upgrade.yaml create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/pre-controller-upgrade.yaml create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/pre-server-upgrade.yaml create mode 100644 pinot-integration-tests/src/test/resources/compat-tests/sample.yaml - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [Issue 6068] Fixing the calls to Helix to throw exception if zk conne… (#6069)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 24147dd [Issue 6068] Fixing the calls to Helix to throw exception if
zk conne… (#6069)
24147dd is described below
commit 24147dd5b7dc188e01af283bd0d025a6cec3527b
Author: Subbu Subramaniam
AuthorDate: Mon Oct 5 17:07:53 2020 -0700
[Issue 6068] Fixing the calls to Helix to throw exception if zk conne…
(#6069)
* [Issue 6068] Fixing the calls to Helix to throw exception if zk
connection is broken
See Issue #6068
* Update zk metadata APIs to use timeout zk timeout
These APIs will ensure that if there is a zk disconnect
we will get an exception after a minimal number of retries.
We can change this to retry once and implement a backoff retry
if needed later on.
Note that the underlying helix library ends up calling the previous
API (as yet), but we will upgrade to a helix version soon that actually
implements these
* Fixing a unit test mock call
* Updating more calls to getChildren
---
.../java/org/apache/pinot/broker/routing/RoutingManager.java | 3 ++-
.../org/apache/pinot/common/metadata/ZKMetadataProvider.java | 10 +++---
.../java/org/apache/pinot/common/utils/CommonConstants.java| 3 +++
.../java/org/apache/pinot/common/utils/helix/HelixHelper.java | 2 +-
.../pinot/controller/helix/core/PinotHelixResourceManager.java | 2 +-
.../pinot/controller/helix/core/rebalance/TableRebalancer.java | 2 +-
.../segment/OfflineReplicaGroupSegmentAssignmentTest.java | 2 +-
.../main/java/org/apache/pinot/tools/UpdateSegmentState.java | 4 +++-
8 files changed, 19 insertions(+), 9 deletions(-)
diff --git
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java
index 8fe3f52..940e40c 100644
---
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java
+++
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/RoutingManager.java
@@ -213,7 +213,8 @@ public class RoutingManager implements ClusterChangeHandler
{
long startTimeMs = System.currentTimeMillis();
List instanceConfigZNRecords =
-_zkDataAccessor.getChildren(_instanceConfigsPath, null,
AccessOption.PERSISTENT);
+_zkDataAccessor.getChildren(_instanceConfigsPath, null,
AccessOption.PERSISTENT,
+CommonConstants.Helix.ZkClient.RETRY_COUNT,
CommonConstants.Helix.ZkClient.RETRY_INTERVAL_MS);
long fetchInstanceConfigsEndTimeMs = System.currentTimeMillis();
// Calculate new enabled and disabled instances
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
b/pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
index 8278777..af5a5d6 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
@@ -31,6 +31,7 @@ import
org.apache.pinot.common.metadata.instance.InstanceZKMetadata;
import org.apache.pinot.common.metadata.segment.LLCRealtimeSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata;
import org.apache.pinot.common.metadata.segment.RealtimeSegmentZKMetadata;
+import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.SchemaUtils;
import org.apache.pinot.common.utils.SegmentName;
import org.apache.pinot.common.utils.StringUtil;
@@ -305,7 +306,8 @@ public class ZKMetadataProvider {
ZkHelixPropertyStore propertyStore, String tableName) {
String offlineTableName =
TableNameBuilder.OFFLINE.tableNameWithType(tableName);
String parentPath =
constructPropertyStorePathForResource(offlineTableName);
-List znRecords = propertyStore.getChildren(parentPath, null,
AccessOption.PERSISTENT);
+List znRecords = propertyStore.getChildren(parentPath, null,
AccessOption.PERSISTENT,
+CommonConstants.Helix.ZkClient.RETRY_COUNT,
CommonConstants.Helix.ZkClient.RETRY_INTERVAL_MS);
if (znRecords != null) {
int numZNRecords = znRecords.size();
List offlineSegmentZKMetadataList = new
ArrayList<>(numZNRecords);
@@ -335,7 +337,8 @@ public class ZKMetadataProvider {
ZkHelixPropertyStore propertyStore, String tableName) {
String realtimeTableName =
TableNameBuilder.REALTIME.tableNameWithType(tableName);
String parentPath =
constructPropertyStorePathForResource(realtimeTableName);
-List znRecords = propertyStore.getChildren(parentPath, null,
AccessOption.PERSISTENT);
+List znRecords = propertyStore.getChildren(parentPath, null,
AccessOption.PERS
[incubator-pinot] branch master updated (e892cb2 -> 40cb64d)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from e892cb2 Enhance DistinctCountThetaSketchAggregationFunction (#6004) add 40cb64d Add list of allowed tables for emitting table level metrics (#6037) No new revisions were added by this update. Summary of changes: .../broker/broker/helix/HelixBrokerStarter.java| 4 ++- .../LiteralOnlyBrokerRequestTest.java | 5 +-- .../pinot/common/metrics/AbstractMetrics.java | 39 +- .../apache/pinot/common/metrics/BrokerMetrics.java | 16 + .../apache/pinot/common/metrics/ServerMetrics.java | 16 + .../apache/pinot/common/utils/CommonConstants.java | 9 +++-- .../org/apache/pinot/server/conf/ServerConf.java | 13 ++-- .../pinot/server/starter/ServerInstance.java | 3 +- .../pinot/spi/utils/builder/TableNameBuilder.java | 9 + .../spi/utils/builder/TableNameBuilderTest.java| 26 --- 10 files changed, 99 insertions(+), 41 deletions(-) copy pinot-common/src/test/java/org/apache/pinot/common/function/FunctionDefinitionRegistryTest.java => pinot-spi/src/test/java/org/apache/pinot/spi/utils/builder/TableNameBuilderTest.java (56%) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Fix CSV reader class path (#5992)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new feeba9e Fix CSV reader class path (#5992)
feeba9e is described below
commit feeba9e3b4eae3501947429451fa39cc21dd8a0f
Author: Subbu Subramaniam
AuthorDate: Wed Sep 9 15:35:03 2020 -0700
Fix CSV reader class path (#5992)
Fixed the classpath of CSV reader in the code.
Introduced comment marker so that we can have comments
in the csv file as well
---
.../apache/pinot/plugin/inputformat/csv/CSVRecordReader.java | 2 ++
.../pinot/plugin/inputformat/csv/CSVRecordReaderConfig.java| 10 ++
.../org/apache/pinot/spi/data/readers/RecordReaderFactory.java | 2 +-
3 files changed, 13 insertions(+), 1 deletion(-)
diff --git
a/pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReader.java
b/pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReader.java
index 43270a3..2802fb0 100644
---
a/pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReader.java
+++
b/pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReader.java
@@ -87,6 +87,8 @@ public class CSVRecordReader implements RecordReader {
} else {
format = format.withHeader(StringUtils.split(csvHeader, delimiter));
}
+ Character commentMarker = config.getCommentMarker();
+ format = format.withCommentMarker(commentMarker);
_format = format;
multiValueDelimiter = config.getMultiValueDelimiter();
}
diff --git
a/pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReaderConfig.java
b/pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReaderConfig.java
index 03c3610..b30c09f 100644
---
a/pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReaderConfig.java
+++
b/pinot-plugins/pinot-input-format/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVRecordReaderConfig.java
@@ -33,6 +33,8 @@ public class CSVRecordReaderConfig implements
RecordReaderConfig {
private char _delimiter = DEFAULT_DELIMITER;
private char _multiValueDelimiter = DEFAULT_MULTI_VALUE_DELIMITER;
+ private Character _commentMarker; // Default is null
+
public String getFileFormat() {
return _fileFormat;
}
@@ -65,6 +67,14 @@ public class CSVRecordReaderConfig implements
RecordReaderConfig {
_multiValueDelimiter = multiValueDelimiter;
}
+ public Character getCommentMarker() {
+return _commentMarker;
+ }
+
+ public void setCommentMarker(Character commentMarker) {
+_commentMarker = commentMarker;
+ }
+
@Override
public String toString() {
return ToStringBuilder.reflectionToString(this,
ToStringStyle.SHORT_PREFIX_STYLE);
diff --git
a/pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/RecordReaderFactory.java
b/pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/RecordReaderFactory.java
index 174decf..4bba3f5 100644
---
a/pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/RecordReaderFactory.java
+++
b/pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/RecordReaderFactory.java
@@ -41,7 +41,7 @@ public class RecordReaderFactory {
private static final String DEFAULT_CSV_RECORD_READER_CLASS =
"org.apache.pinot.plugin.inputformat.csv.CSVRecordReader";
private static final String DEFAULT_CSV_RECORD_READER_CONFIG_CLASS =
- "org.apache.pinot.csv.data.readers.CSVRecordReaderConfig";
+ "org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig";
private static final String DEFAULT_JSON_RECORD_READER_CLASS =
"org.apache.pinot.plugin.inputformat.json.JSONRecordReader";
private static final String DEFAULT_THRIFT_RECORD_READER_CLASS =
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Add compatibility verifier scripts (#4854) (#5911)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new f3ed95c Add compatibility verifier scripts (#4854) (#5911)
f3ed95c is described below
commit f3ed95cfd5d8733a2dda8354e80f03fd7691523f
Author: Sailesh Nankani <30644225+saileshnank...@users.noreply.github.com>
AuthorDate: Wed Sep 2 16:31:45 2020 -0400
Add compatibility verifier scripts (#4854) (#5911)
* Add compatibility verifier scripts (#4854)
Adds 3 scripts to take 2 commit hashes and verify compatibility across
these.
- build-release.sh
- comp-verifier.sh
- control-service.sh
Starts all components in v1 version. Upgrades controller first, then the
broker, and finally the server.
* Format and address review comments
* Add error handling and comments
* Address further PR comments
* Fix additional PR comments
---
compatibility-verifier/compCheck.sh | 177
1 file changed, 177 insertions(+)
diff --git a/compatibility-verifier/compCheck.sh
b/compatibility-verifier/compCheck.sh
new file mode 100755
index 000..af81744
--- /dev/null
+++ b/compatibility-verifier/compCheck.sh
@@ -0,0 +1,177 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# A script that does rolling upgrade of Pinot components
+# from one version to the other given 2 commit hashes. It first builds
+# Pinot in the 2 given directories and then upgrades in the following order:
+# Controller -> Broker -> Server
+
+# get a temporary directory in case the workingDir is not provided by user
+TMP_DIR=$(mktemp -d 2>/dev/null || mktemp -d -t 'mytmpdir')
+
+# get usage of the script
+function usage() {
+ command=$1
+ echo "Usage: $command olderCommit newerCommit [workingDir]"
+ exit 1
+}
+
+# cleanup the temporary directory when exiting the script
+function cleanup() {
+ if [ -n "$TMP_DIR" ] && [ -d "$TMP_DIR" ] && [ "$workingDir" = "$TMP_DIR" ]
; then
+echo "The temporary directory $TMP_DIR needs to be cleaned up."
+ fi
+}
+
+# This function builds Pinot given a specific commit hash and target directory
+function checkoutAndBuild() {
+ commitHash=$1
+ targetDir=$2
+
+ pushd "$targetDir" || exit 1
+ git init
+ git remote add origin https://github.com/apache/incubator-pinot
+ git fetch --depth 1 origin "$commitHash"
+ git checkout FETCH_HEAD
+ mvn install package -DskipTests -Pbin-dist
+ popd || exit 1
+}
+
+# Given a component and directory, start that version of the specific
component
+function startService() {
+ serviceName=$1
+ dirName=$2
+ # Upon start, save the pid of the process for a component into a file in
/tmp/{component}.pid, which is then used to stop it
+ pushd "$dirName"/pinot-tools/target/pinot-tools-pkg/bin || exit 1
+ if [ "$serviceName" = "zookeeper" ]; then
+sh -c 'echo $$ > $0/zookeeper.pid; exec ./pinot-admin.sh StartZookeeper'
"${dirName}" &
+ elif [ "$serviceName" = "controller" ]; then
+sh -c 'echo $$ > $0/controller.pid; exec ./pinot-admin.sh StartController'
"${dirName}" &
+ elif [ "$serviceName" = "broker" ]; then
+sh -c 'echo $$ > $0/broker.pid; exec ./pinot-admin.sh StartBroker'
"${dirName}" &
+ elif [ "$serviceName" = "server" ]; then
+sh -c 'echo $$ > $0/server.pid; exec ./pinot-admin.sh StartServer'
"${dirName}" &
+ fi
+ popd || exit 1
+}
+
+# Given a component, check if it known to be running and stop that specific
component
+function stopService() {
+ serviceName=$1
+ dirName=$2
+ if [ -f "${dirName}/${serviceName}".pid ]; then
+servicePid=$(<"${dirName}/${serviceName}".pid)
+rm "${dirName}/${serviceName}".pid
+if [ -n "$se
[incubator-pinot] branch master updated (debadaa -> 63a4fd4)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from debadaa [TE] enchance anomaly api to propagate feedback (#5823) add 63a4fd4 Clarifications in realtime provisioning helper (#5838) No new revisions were added by this update. Summary of changes: .../tools/admin/command/RealtimeProvisioningHelperCommand.java | 7 --- .../apache/pinot/tools/realtime/provisioning/MemoryEstimator.java | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Improvements to RealtimeProvisioningHelper command (#5737)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 540853d Improvements to RealtimeProvisioningHelper command (#5737) 540853d is described below commit 540853d786c9e4b4bb27bf4bbc6fa73837ff1bc3 Author: Subbu Subramaniam AuthorDate: Fri Jul 24 20:18:37 2020 -0700 Improvements to RealtimeProvisioningHelper command (#5737) * Improvements to RealtimepRovisioningHelper command - Fixed a bug where we were looking for push frequency setting in the table config Push frequency is not usually set in realtime table, so changed that to be an optional argument instead. - Instead of push frequency, the user may specify retentionHours, meaning the number of hours of most recent data that will be queried most often. - Added the amount of total mapped memory to the output - Added the number of segments queried per host to the output. - Added a link to the documentation to the output. - Instead of accepting the time taken to consume the segment, introduced an argument to accept the ingestion rate per partition. This is determined easier when looking at metrics for the topic that we need to ingest, and is more convenient when an offline segment is being provided as a sample - Added the command line arguments printed with the output so that it is easy for us to debug when we get questions from the community. TODO Next: Can we compute a score and just recommend a configuration? We know we want to minimize num segments scanned per host, num hosts, segment size (to fit memory). Issue #5588 Here is a sample output: RealtimeProvisioningHelperCommand -tableConfigFile /Users/ssubrama/tmp/samza/realtimeTableConfig.json -numPartitions 16 -pushFrequency null -numHosts 8,6,10 -numHours 6,12,18,24 -sampleCompletedSegmentDir /Users/ssubrama/tmp/samza/TestSamzaAnalyticsFeatures_159341148_159350034_0/ -ingestionRate 100 -maxUsableHostMemory 48G -retentionHours 72 Note: * Table retention and push frequency ignored for determining retentionHours * See https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime Memory used per host (Active/Mapped) numHosts --> 6 |8 |10 | numHours 6 > 5.05G/19.49G|3.37G/12.99G|3.37G/12.99G| 12 > 5.89G/20.33G|3.93G/13.55G|3.93G/13.55G| 18 > 6.73G/21.49G|4.48G/14.33G|4.48G/14.33G| 24 > 7.56G/22G |5.04G/14.66G|5.04G/14.66G| Optimal segment size numHosts --> 6 |8 |10 | numHours 6 > 111.98M |111.98M |111.98M | 12 > 223.96M |223.96M |223.96M | 18 > 335.94M |335.94M |335.94M | 24 > 447.92M |447.92M |447.92M | Consuming memory numHosts --> 6 |8 |10 | numHours 6 > 1.45G |987.17M |987.17M | 12 > 2.61G |1.74G |1.74G | 18 > 3.77G |2.52G |2.52G | 24 > 4.94G |3.29G |3.29G | Number of segments queried per host numHosts --> 6 |8 |10 | numHours 6 > 12 |12 |12 | 12 > 6 |6 |6 | 18 > 4 |4 |4 | 24 > 3 |3 |3 | * Addressed review comments --- .../routing/timeboundary/TimeBoundaryManager.java | 2 +- .../apache/pinot/common/utils/CommonConstants.java | 7 + .../controller/util/SegmentIntervalUtils.java | 9 +- .../name/NormalizedDateSegmentNameGenerator.java | 3 +- .../command/RealtimeProvisioningHelperCommand.java | 100 ++ .../realtime/provisioning/MemoryEstimator.java | 144 - 6 files changed, 168 insertions(+), 97 deletions(-) diff --git a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java index 5e4b019..7462913 100644 --- a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/timeboundary/TimeBoundaryManager.java +++ b/pinot-broke
[incubator-pinot] branch master updated (ee72faa -> dab1c2d)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from ee72faa update data generator schema json with dataTimeSpec (#5739) add dab1c2d [Deepstore by-passing]Introduce a subclasses SplitSegmentCommitter which will proceeds to commit even if the segment upload fails. (#5700) No new revisions were added by this update. Summary of changes: .../apache/pinot/common/utils/CommonConstants.java | 1 + .../realtime/PinotLLCRealtimeSegmentManager.java | 8 ++-- .../PinotLLCRealtimeSegmentManagerTest.java| 3 +- .../helix/core/realtime/SegmentCompletionTest.java | 3 +- .../realtime/LLRealtimeSegmentDataManager.java | 24 +++ .../realtime/PeerSchemeSplitSegmentCommitter.java | 48 ++ .../manager/realtime/PinotFSSegmentUploader.java | 2 + .../manager/realtime/SegmentCommitterFactory.java | 37 + .../manager/realtime/SplitSegmentCommitter.java| 14 ++- .../segment/index/loader/IndexLoadingConfig.java | 7 .../realtime/LLRealtimeSegmentDataManagerTest.java | 2 + 11 files changed, 117 insertions(+), 32 deletions(-) create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/PeerSchemeSplitSegmentCommitter.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Fix ORC Record reader to ignore extra fields (#5645)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch hotfix-orc
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit e1ae65682f7a54c2aac5f01be773691918452f8e
Author: Subbu Subramaniam
AuthorDate: Mon Jul 13 15:36:15 2020 -0700
Fix ORC Record reader to ignore extra fields (#5645)
* Fix ORC Record reader to ignore extra fields
Fixing an issue introduced in PR #5267
We should not be validating the type of fields that we don't care about.
Cleaned up the messages and exceptions thrown so that we know which
field is the problematic one.
* Update
pinot-plugins/pinot-input-format/pinot-orc/src/main/java/org/apache/pinot/plugin/inputformat/orc/ORCRecordReader.java
Co-authored-by: Xiaotian (Jackie) Jiang
<1751+jackie-ji...@users.noreply.github.com>
Co-authored-by: Xiaotian (Jackie) Jiang
<1751+jackie-ji...@users.noreply.github.com>
---
.../plugin/inputformat/orc/ORCRecordReader.java| 24 +++---
.../inputformat/orc/ORCRecordReaderTest.java | 2 +-
2 files changed, 13 insertions(+), 13 deletions(-)
diff --git
a/pinot-plugins/pinot-input-format/pinot-orc/src/main/java/org/apache/pinot/plugin/inputformat/orc/ORCRecordReader.java
b/pinot-plugins/pinot-input-format/pinot-orc/src/main/java/org/apache/pinot/plugin/inputformat/orc/ORCRecordReader.java
index dbc3f68..b792c57 100644
---
a/pinot-plugins/pinot-input-format/pinot-orc/src/main/java/org/apache/pinot/plugin/inputformat/orc/ORCRecordReader.java
+++
b/pinot-plugins/pinot-input-format/pinot-orc/src/main/java/org/apache/pinot/plugin/inputformat/orc/ORCRecordReader.java
@@ -97,8 +97,8 @@ public class ORCRecordReader implements RecordReader {
if (category == TypeDescription.Category.LIST) {
// Multi-value field
TypeDescription.Category childCategory =
fieldType.getChildren().get(0).getCategory();
- Preconditions.checkState(isSupportedSingleValueType(childCategory),
"Illegal multi-value field type: %s",
- childCategory);
+ Preconditions.checkState(isSupportedSingleValueType(childCategory),
"Illegal multi-value field type: %s (field %s)",
+ childCategory, field);
// NOTE: LIST is stored as 2 vectors
int fieldId = fieldType.getId();
orcReaderInclude[fieldId] = true;
@@ -108,10 +108,10 @@ public class ORCRecordReader implements RecordReader {
List children = fieldType.getChildren();
TypeDescription.Category keyCategory = children.get(0).getCategory();
Preconditions
- .checkState(isSupportedSingleValueType(keyCategory), "Illegal
map key field type: %s", keyCategory);
+ .checkState(isSupportedSingleValueType(keyCategory), "Illegal
map key field type: %s (field %s)", keyCategory, field);
TypeDescription.Category valueCategory =
children.get(1).getCategory();
Preconditions
- .checkState(isSupportedSingleValueType(valueCategory), "Illegal
map value field type: %s", valueCategory);
+ .checkState(isSupportedSingleValueType(valueCategory), "Illegal
map value field type: %s (field %s)", valueCategory, field);
// NOTE: MAP is stored as 3 vectors
int fieldId = fieldType.getId();
orcReaderInclude[fieldId] = true;
@@ -120,11 +120,11 @@ public class ORCRecordReader implements RecordReader {
} else {
// Single-value field
Preconditions
- .checkState(isSupportedSingleValueType(category), "Illegal
single-value field type: %s", category);
+ .checkState(isSupportedSingleValueType(category), "Illegal
single-value field type: %s (field %s)", category, field);
orcReaderInclude[fieldType.getId()] = true;
}
+_includeOrcFields[i] = true;
}
- _includeOrcFields[i] = true;
}
_orcRecordReader = orcReader.rows(new
Reader.Options().include(orcReaderInclude));
@@ -186,7 +186,7 @@ public class ORCRecordReader implements RecordReader {
int length = (int) listColumnVector.lengths[rowId];
List values = new ArrayList<>(length);
for (int j = 0; j < length; j++) {
-Object value = extractSingleValue(listColumnVector.child, offset +
j, childCategory);
+Object value = extractSingleValue(field, listColumnVector.child,
offset + j, childCategory);
// NOTE: Only keep non-null values
// TODO: Revisit
if (value != null) {
@@ -216,8 +216,8 @@ public class ORCRecordReader implements RecordReader {
Map map = new HashMap<>();
for (int j = 0; j < length; j++) {
int childRowId = offset + j;
-Object key = extractSingleValue(map
[incubator-pinot] branch hotfix-orc created (now e1ae656)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch hotfix-orc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at e1ae656 Fix ORC Record reader to ignore extra fields (#5645) This branch includes the following new commits: new e1ae656 Fix ORC Record reader to ignore extra fields (#5645) The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (24b2cba -> 85ec229)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 24b2cba Add Streaming Server Endpoint (#5599) add 85ec229 Fix ORC Record reader to ignore extra fields (#5645) No new revisions were added by this update. Summary of changes: .../plugin/inputformat/orc/ORCRecordReader.java| 24 +++--- .../inputformat/orc/ORCRecordReaderTest.java | 2 +- 2 files changed, 13 insertions(+), 13 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (4d37dee -> 8af7d38)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 4d37dee Added direnv config file to gitignore (#5644) add 8af7d38 [Part 4] Deep-store bypass for LLC: Add a peer to peer segment fetcher. (#5336) No new revisions were added by this update. Summary of changes: .../broker/broker/BrokerAdminApiApplication.java | 3 +- .../apache/pinot/common/utils/CommonConstants.java | 2 + .../common/utils/FileUploadDownloadClient.java | 4 +- .../org/apache/pinot/common/utils/SchemaUtils.java | 6 +- .../common/utils/config/TableConfigUtils.java | 7 +- .../common/utils/fetcher/BaseSegmentFetcher.java | 22 .../pinot/common/utils/fetcher/SegmentFetcher.java | 7 ++ .../utils/fetcher/SegmentFetcherFactory.java | 11 +- .../common/utils/webhdfs/WebHdfsV1Client.java | 3 +- .../common/utils/config/TableConfigSerDeTest.java | 11 +- .../utils/fetcher/SegmentFetcherFactoryTest.java | 12 ++- .../apache/pinot/controller/ControllerConf.java| 7 +- .../apache/pinot/controller/ControllerStarter.java | 2 +- .../api/ControllerAdminApiApplication.java | 21 ++-- .../core/data/manager/BaseTableDataManager.java| 5 +- .../pinot/core/data/manager/TableDataManager.java | 3 +- .../manager/offline/TableDataManagerProvider.java | 5 +- .../realtime/LLRealtimeSegmentDataManager.java | 2 +- .../manager/realtime/RealtimeTableDataManager.java | 83 +-- .../pinot/core/util/PeerServerSegmentFinder.java | 104 ++ .../ServerSegmentCompletionProtocolHandler.java| 4 +- .../data/manager/BaseTableDataManagerTest.java | 4 +- .../SegmentGenerationWithNullValueVectorTest.java | 3 +- .../core/util/PeerServerSegmentFinderTest.java | 117 + .../pinot/query/executor/QueryExecutorTest.java| 3 +- .../org/apache/pinot/minion/MinionStarter.java | 3 +- .../server/starter/helix/AdminApiApplication.java | 3 +- .../starter/helix/HelixInstanceDataManager.java| 2 +- .../apache/pinot/server/api/BaseResourceTest.java | 3 +- .../pinot/tools/perf/PerfBenchmarkDriver.java | 2 +- .../PinotServiceManagerAdminApiApplication.java| 3 +- 31 files changed, 402 insertions(+), 65 deletions(-) create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/util/PeerServerSegmentFinder.java create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/util/PeerServerSegmentFinderTest.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (9678035 -> 72311af)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 9678035 SQL Compilation Fixes (#5643) add 72311af Refactor how Pinot controller stores segment download url in Zookeeper to deal with peer uri format (#5639) No new revisions were added by this update. Summary of changes: .../apache/pinot/common/utils/CommonConstants.java | 1 + .../resources/LLCSegmentCompletionHandlers.java| 1 + .../realtime/PinotLLCRealtimeSegmentManager.java | 24 +-- .../core/realtime/SegmentCompletionManager.java| 10 ++ .../PinotLLCRealtimeSegmentManagerTest.java| 36 ++ .../helix/core/realtime/SegmentCompletionTest.java | 31 +-- 6 files changed, 91 insertions(+), 12 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Add segment encryption on Controller based on table config (#5617)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 40a3152 Add segment encryption on Controller based on table config
(#5617)
40a3152 is described below
commit 40a31529213efa7a5d03eb4e0ca020aa4045e0ca
Author: Sajjad
AuthorDate: Wed Jul 1 13:23:24 2020 -0700
Add segment encryption on Controller based on table config (#5617)
* Add segment encryption on Controller based on table config
* Applied PR's comments
* Addressed PR's comments
* Add license header + equality check on crypters
* Addressed PR's comments
* Fix log message
Co-authored-by: Subbu Subramaniam
Co-authored-by: Subbu Subramaniam
---
.../common/utils/config/TableConfigUtils.java | 10 +-
.../pinot/common/utils/helix/TableCache.java | 4 +
.../common/utils/config/TableConfigSerDeTest.java | 6 +-
.../PinotSegmentUploadDownloadRestletResource.java | 122
.../pinot/controller/api/upload/ZKOperator.java| 11 +-
.../helix/core/PinotHelixResourceManager.java | 12 ++
...otSegmentUploadDownloadRestletResourceTest.java | 125 +
.../controller/api/upload/ZKOperatorTest.java | 11 +-
.../SegmentsValidationAndRetentionConfig.java | 13 ++-
.../apache/pinot/spi/crypt/NoOpPinotCrypter.java | 16 ++-
.../spi/utils/builder/TableConfigBuilder.java | 7 ++
11 files changed, 270 insertions(+), 67 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
index 45610fa..2ee7b47 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
@@ -132,8 +132,8 @@ public class TableConfigUtils {
}
return new TableConfig(tableName, tableType, validationConfig,
tenantConfig, indexingConfig, customConfig,
-quotaConfig, taskConfig, routingConfig, queryConfig,
instanceAssignmentConfigMap, fieldConfigList,
-upsertConfig, ingestionConfig);
+quotaConfig, taskConfig, routingConfig, queryConfig,
instanceAssignmentConfigMap, fieldConfigList, upsertConfig,
+ingestionConfig);
}
public static ZNRecord toZNRecord(TableConfig tableConfig)
@@ -228,8 +228,10 @@ public class TableConfigUtils {
}
String peerSegmentDownloadScheme =
validationConfig.getPeerSegmentDownloadScheme();
if (peerSegmentDownloadScheme != null) {
-if (!"http".equalsIgnoreCase(peerSegmentDownloadScheme) &&
!"https".equalsIgnoreCase(peerSegmentDownloadScheme)) {
- throw new IllegalStateException("Invalid value '" +
peerSegmentDownloadScheme + "' for peerSegmentDownloadScheme. Must be one of
http nor https" );
+if (!"http".equalsIgnoreCase(peerSegmentDownloadScheme) && !"https"
+.equalsIgnoreCase(peerSegmentDownloadScheme)) {
+ throw new IllegalStateException("Invalid value '" +
peerSegmentDownloadScheme
+ + "' for peerSegmentDownloadScheme. Must be one of http nor
https");
}
}
}
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/TableCache.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/TableCache.java
index 474adbf..6f46ac2 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/TableCache.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/TableCache.java
@@ -80,6 +80,10 @@ public class TableCache {
return columnName;
}
+ public TableConfig getTableConfig(String tableName) {
+return _tableConfigChangeListener._tableConfigMap.get(tableName);
+ }
+
class TableConfigChangeListener implements IZkChildListener, IZkDataListener
{
Map _tableConfigMap = new ConcurrentHashMap<>();
diff --git
a/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
b/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
index af5e5c0..f877c3f 100644
---
a/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
+++
b/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
@@ -248,8 +248,10 @@ public class TableConfigSerDeTest {
{
// with SegmentsValidationAndRetentionConfig
TableConfig tableConfig =
tableConfigBuilder.setPeerSegmentDownloadScheme("http").build();
-
checkSegmentsValida
[incubator-pinot] branch master updated: Fix backward incompatibility in StreamFactoryConsumerProvider (#5557)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 5e532ec Fix backward incompatibility in StreamFactoryConsumerProvider
(#5557)
5e532ec is described below
commit 5e532eccd8f0898b0b13eef32b9f5335682a037e
Author: Subbu Subramaniam
AuthorDate: Fri Jun 12 17:32:32 2020 -0700
Fix backward incompatibility in StreamFactoryConsumerProvider (#5557)
* Fix backward incompatibility in StreamFactoryConsumerProvider
PR #5542 introduced a backward incompatibility in this class that
would cause existing stream implementations to break when they use
pinot 0.5.0. This commit re-introduces the deleted method
* Eliminated a new factory provider for offset parser
Changed the StreamConsumerFactory to also provide the offset factory.
This is much easier way to keep backward compat and reduces complexity.
---
.../core/realtime/PinotLLCRealtimeSegmentManager.java | 6 --
.../helix/core/realtime/SegmentCompletionManager.java | 2 +-
.../realtime/HLRealtimeSegmentDataManager.java| 2 +-
.../realtime/LLRealtimeSegmentDataManager.java| 5 +++--
.../impl/fakestream/FakeStreamConsumerFactory.java| 2 +-
.../pinot/core/realtime/stream/StreamConfigTest.java | 2 --
.../kafka20/KafkaPartitionLevelConsumerTest.java | 2 +-
.../pinot/spi/stream/PartitionCountFetcher.java | 2 +-
.../pinot/spi/stream/PartitionOffsetFetcher.java | 2 +-
.../org/apache/pinot/spi/stream/StreamConfig.java | 12
.../pinot/spi/stream/StreamConsumerFactory.java | 4
.../spi/stream/StreamConsumerFactoryProvider.java | 19 +--
12 files changed, 18 insertions(+), 42 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
index 32b6aea..9ea0613 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
@@ -508,7 +508,8 @@ public class PinotLLCRealtimeSegmentManager {
int numPartitions, int numReplicas) {
String realtimeTableName = tableConfig.getTableName();
String segmentName = newLLCSegmentName.getSegmentName();
-StreamPartitionMsgOffsetFactory offsetFactory =
StreamConsumerFactoryProvider.createOffsetFactory(streamConfig);
+StreamPartitionMsgOffsetFactory offsetFactory =
+
StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
StreamPartitionMsgOffset startOffset =
offsetFactory.create(committingSegmentDescriptor.getNextOffset());
LOGGER
.info("Creating segment ZK metadata for new CONSUMING segment: {} with
start offset: {} and creation time: {}",
@@ -830,7 +831,8 @@ public class PinotLLCRealtimeSegmentManager {
Map> instanceStatesMap =
idealState.getRecord().getMapFields();
long currentTimeMs = getCurrentTimeMs();
-StreamPartitionMsgOffsetFactory offsetFactory =
StreamConsumerFactoryProvider.createOffsetFactory(streamConfig);
+StreamPartitionMsgOffsetFactory offsetFactory =
+
StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
// Get the latest segment ZK metadata for each partition
Map latestSegmentZKMetadataMap =
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java
index 1424b17..3353a6b 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java
@@ -124,7 +124,7 @@ public class SegmentCompletionManager {
TableConfig tableConfig =
_segmentManager.getTableConfig(TableNameBuilder.REALTIME.tableNameWithType(rawTableName));
PartitionLevelStreamConfig streamConfig =
new PartitionLevelStreamConfig(tableConfig.getTableName(),
tableConfig.getIndexingConfig().getStreamConfigs());
-return StreamConsumerFactoryProvider.createOffsetFactory(streamConfig);
+return
StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
}
// We need to make sure that we never create multiple FSMs for the same
segment
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
b/pinot-core/src/
[incubator-pinot] branch master updated: Fixed backward incompatibility for existing stream implementations (#5549)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 7326d4a Fixed backward incompatibility for existing stream
implementations (#5549)
7326d4a is described below
commit 7326d4ad7a7db1d07d760f6ace1b59fa4740a8d0
Author: Subbu Subramaniam
AuthorDate: Fri Jun 12 10:46:02 2020 -0700
Fixed backward incompatibility for existing stream implementations (#5549)
PR #5542 introduced incompatibility in the interfaces for existing
stream implementations (like eventhub or other kafka implementations)
Added default implementations so that the other streams can still
compile with pinot.
---
.../data/manager/realtime/LLRealtimeSegmentDataManager.java | 3 +--
.../realtime/impl/fakestream/FakeStreamMessageBatch.java | 6 +-
.../plugin/stream/kafka09/SimpleConsumerMessageBatch.java| 7 ++-
.../pinot/plugin/stream/kafka20/KafkaMessageBatch.java | 7 ++-
.../main/java/org/apache/pinot/spi/stream/MessageBatch.java | 12 +++-
.../org/apache/pinot/spi/stream/StreamMetadataProvider.java | 7 +--
6 files changed, 34 insertions(+), 8 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
index 05f9340..6dcef0a 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
@@ -66,7 +66,6 @@ import
org.apache.pinot.spi.config.table.SegmentPartitionConfig;
import org.apache.pinot.spi.config.table.TableConfig;
import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.spi.data.readers.GenericRow;
-import org.apache.pinot.spi.plugin.PluginManager;
import org.apache.pinot.spi.stream.MessageBatch;
import org.apache.pinot.spi.stream.PartitionLevelConsumer;
import org.apache.pinot.spi.stream.PartitionLevelStreamConfig;
@@ -506,7 +505,7 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
realtimeRowsDroppedMeter);
}
- _currentOffset =
messagesAndOffsets.getNextStreamMessageOffsetAtIndex(index);
+ _currentOffset =
messagesAndOffsets.getNextStreamParitionMsgOffsetAtIndex(index);
_numRowsIndexed = _realtimeSegment.getNumDocsIndexed();
_numRowsConsumed++;
streamMessageCount++;
diff --git
a/pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/fakestream/FakeStreamMessageBatch.java
b/pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/fakestream/FakeStreamMessageBatch.java
index 650d75a..59dd753 100644
---
a/pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/fakestream/FakeStreamMessageBatch.java
+++
b/pinot-core/src/test/java/org/apache/pinot/core/realtime/impl/fakestream/FakeStreamMessageBatch.java
@@ -52,7 +52,11 @@ public class FakeStreamMessageBatch implements
MessageBatch {
return _messageBytes.get(index).length;
}
- public StreamPartitionMsgOffset getNextStreamMessageOffsetAtIndex(int index)
{
+ public long getNextStreamMessageOffsetAtIndex(int index) {
+throw new UnsupportedOperationException("This method is deprecated");
+ }
+
+ public StreamPartitionMsgOffset getNextStreamParitionMsgOffsetAtIndex(int
index) {
return new LongMsgOffset(_messageOffsets.get(index) + 1);
}
}
diff --git
a/pinot-plugins/pinot-stream-ingestion/pinot-kafka-0.9/src/main/java/org/apache/pinot/plugin/stream/kafka09/SimpleConsumerMessageBatch.java
b/pinot-plugins/pinot-stream-ingestion/pinot-kafka-0.9/src/main/java/org/apache/pinot/plugin/stream/kafka09/SimpleConsumerMessageBatch.java
index b2125ff..ab333de 100644
---
a/pinot-plugins/pinot-stream-ingestion/pinot-kafka-0.9/src/main/java/org/apache/pinot/plugin/stream/kafka09/SimpleConsumerMessageBatch.java
+++
b/pinot-plugins/pinot-stream-ingestion/pinot-kafka-0.9/src/main/java/org/apache/pinot/plugin/stream/kafka09/SimpleConsumerMessageBatch.java
@@ -51,7 +51,12 @@ public class SimpleConsumerMessageBatch implements
MessageBatch {
return messageList.get(index).message().payloadSize();
}
- public StreamPartitionMsgOffset getNextStreamMessageOffsetAtIndex(int index)
{
+ @Override
+ public long getNextStreamMessageOffsetAtIndex(int index) {
+throw new UnsupportedOperationException("This method is deprecated");
+ }
+
+ public StreamPartitionMsgOffset getNextStreamParitionMsgOffsetAtIndex(int
index) {
return new LongMsgOffset(messageList.get(index).nextOffset());
}
}
diff --git
a/pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/p
[incubator-pinot] branch master updated (55b4ddc -> f1f1c41)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 55b4ddc Change default segment load mode to MMAP. (#5539) add f1f1c41 Changed the stream and metadata interface (#5542) No new revisions were added by this update. Summary of changes: .../segment/LLCRealtimeSegmentZKMetadata.java | 29 +++- .../protocols/SegmentCompletionProtocol.java | 1 - .../realtime/PinotLLCRealtimeSegmentManager.java | 28 +-- .../core/realtime/SegmentCompletionManager.java| 7 ++- .../PinotLLCRealtimeSegmentManagerTest.java| 54 +++--- .../helix/core/realtime/SegmentCompletionTest.java | 2 +- .../helix/core/retention/RetentionManagerTest.java | 5 +- .../realtime/LLRealtimeSegmentDataManager.java | 12 ++--- .../realtime/LLRealtimeSegmentDataManagerTest.java | 6 +-- .../fakestream/FakePartitionLevelConsumer.java | 22 ++--- .../impl/fakestream/FakeStreamConfigUtils.java | 10 ++-- .../impl/fakestream/FakeStreamConsumerFactory.java | 8 ++-- .../impl/fakestream/FakeStreamMessageBatch.java| 6 ++- .../fakestream/FakeStreamMetadataProvider.java | 7 ++- .../kafka09/KafkaPartitionLevelConsumer.java | 12 - .../kafka09/KafkaStreamMetadataProvider.java | 11 - .../stream/kafka09/SimpleConsumerMessageBatch.java | 6 ++- .../kafka09/KafkaPartitionLevelConsumerTest.java | 9 ++-- .../plugin/stream/kafka20/KafkaMessageBatch.java | 6 ++- .../kafka20/KafkaPartitionLevelConsumer.java | 10 .../kafka20/KafkaStreamMetadataProvider.java | 15 -- .../kafka20/KafkaPartitionLevelConsumerTest.java | 26 +++ .../pinot/spi/stream/LongMsgOffsetFactory.java | 10 .../org/apache/pinot/spi/stream/MessageBatch.java | 2 +- .../pinot/spi/stream/PartitionLevelConsumer.java | 20 +++- .../pinot/spi/stream/PartitionOffsetFetcher.java | 6 +-- .../pinot/spi/stream/StreamMetadataProvider.java | 6 ++- .../pinot/spi/stream/StreamPartitionMsgOffset.java | 29 +--- .../stream/StreamPartitionMsgOffsetFactory.java| 24 +++--- 29 files changed, 246 insertions(+), 143 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Changed the segment commit protocol to send/receive streamPartitionMs… (#5486)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 056c930 Changed the segment commit protocol to send/receive
streamPartitionMs… (#5486)
056c930 is described below
commit 056c930c6e62bbcd4cd3e8c659e16e815532dc4c
Author: Subbu Subramaniam
AuthorDate: Wed Jun 3 13:01:24 2020 -0700
Changed the segment commit protocol to send/receive streamPartitionMs…
(#5486)
* Changed the segment commit protocol to send/receive
streamPartitionMsgOffset
Updated the segment commit protocol so that new element
streamPartitionMsgOffset
is populated in requests (as request parameters) and in response (as JSON
string element)
The server side has been modified to send the 'streamPartitionMsgOffset' as
well as we the
'offset' parameters to the controller. The controller looks for and prefers
streamPartitionMsgOffset
but falls back to offset if the streamPartitionMsgOffset is not there.
The controller side, in the repsonse message, populates both of the
elements, and the server on
the receiver side does likewise -- preferring streamPartitionMsgOffset.
All callers into the protocol module have been modified to NOT set the
offset field. Instead,
only set the streamPartitionMsgOffset field. The 'offset' value will be
derived from
streamPartitionMsgOffset.
Added a test to make sure that the controller always generates both
elements. Such a test was not
possible in the server side at this time, so verified manually.
Manually ran LLCClusterIntergrationTest by disabling populating
`streamPartitionMsgOffset` on the
server side (old server/new controller) and on the controller respons side
(new server/old controller)
Issue #5359
* Addressed review comments
---
.../protocols/SegmentCompletionProtocol.java | 76 +++--
.../resources/LLCSegmentCompletionHandlers.java| 93 --
.../api/SegmentCompletionProtocolDeserTest.java| 4 +-
.../helix/core/realtime/SegmentCompletionTest.java | 311 +++--
.../realtime/LLRealtimeSegmentDataManager.java | 2 +-
.../realtime/LLRealtimeSegmentDataManagerTest.java | 70 ++---
.../tests/SegmentCompletionIntegrationTest.java| 5 +-
.../pinot/spi/stream/StreamPartitionMsgOffset.java | 9 +
8 files changed, 340 insertions(+), 230 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java
b/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java
index 3150504..8af8dbf 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/protocols/SegmentCompletionProtocol.java
@@ -122,6 +122,7 @@ public class SegmentCompletionProtocol {
public static final String PARAM_SEGMENT_LOCATION = "location";
public static final String PARAM_SEGMENT_NAME = "name";
public static final String PARAM_OFFSET = "offset";
+ public static final String PARAM_STREAM_PARTITION_MSG_OFFSET =
"streamPartitionMsgOffset";
public static final String PARAM_INSTANCE_ID = "instance";
public static final String PARAM_MEMORY_USED_BYTES = "memoryUsedBytes";
public static final String PARAM_SEGMENT_SIZE_BYTES = "segmentSizeBytes";
@@ -189,7 +190,10 @@ public class SegmentCompletionProtocol {
+ (_params.getSegmentSizeBytes() <= 0 ? ""
: ("&" + PARAM_SEGMENT_SIZE_BYTES + "=" +
_params.getSegmentSizeBytes())) + (_params.getNumRows() <= 0 ? ""
: ("&" + PARAM_ROW_COUNT + "=" + _params.getNumRows())) +
(_params.getSegmentLocation() == null ? ""
- : ("&" + PARAM_SEGMENT_LOCATION + "=" +
_params.getSegmentLocation()));
+ : ("&" + PARAM_SEGMENT_LOCATION + "=" +
_params.getSegmentLocation()))
+ + (_params.getStreamPartitionMsgOffset() == null ? ""
+ : ("&" + PARAM_STREAM_PARTITION_MSG_OFFSET + "=" +
_params.getStreamPartitionMsgOffset()))
+ ;
}
public static class Params {
@@ -204,6 +208,7 @@ public class SegmentCompletionProtocol {
private String _segmentLocation;
private long _memoryUsedBytes;
private long _segmentSizeBytes;
+ private String _streamPartitionMsgOffset;
public Params() {
_offset = -1L;
@@ -216,6 +221,7 @@ public class SegmentCompletionProtocol {
_segmentLocation = null;
_memoryUsedBytes = MEMORY_USED_BYTES_DEFAULT
[incubator-pinot] branch master updated: Add a new table config field for peer segment download. (#5478)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 8a3eb43 Add a new table config field for peer segment download.
(#5478)
8a3eb43 is described below
commit 8a3eb4315df59d6d73a4ec11308d30eab1c51042
Author: Ting Chen
AuthorDate: Tue Jun 2 14:00:47 2020 -0700
Add a new table config field for peer segment download. (#5478)
* Add a new table config field for peer segment download.
* Address feedbacks.
* Update
pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
Co-authored-by: Subbu Subramaniam
* Update
pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
Co-authored-by: Subbu Subramaniam
Co-authored-by: Subbu Subramaniam
---
.../pinot/common/utils/config/TableConfigUtils.java| 18 ++
.../common/utils/config/TableConfigSerDeTest.java | 11 +++
.../table/SegmentsValidationAndRetentionConfig.java| 11 +++
.../pinot/spi/utils/builder/TableConfigBuilder.java| 7 +++
4 files changed, 47 insertions(+)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
index d93b24b..b7b7d52 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/config/TableConfigUtils.java
@@ -181,9 +181,15 @@ public class TableConfigUtils {
* Validates the table config with the following rules:
*
* Text index column must be raw
+ * peerSegmentDownloadScheme in ValidationConfig must be http or
https
*
*/
public static void validate(TableConfig tableConfig) {
+validateFieldConfigList(tableConfig);
+validateValidationConfig(tableConfig);
+ }
+
+ private static void validateFieldConfigList(TableConfig tableConfig) {
List fieldConfigList = tableConfig.getFieldConfigList();
if (fieldConfigList != null) {
List noDictionaryColumns =
tableConfig.getIndexingConfig().getNoDictionaryColumns();
@@ -201,4 +207,16 @@ public class TableConfigUtils {
}
}
}
+
+ private static void validateValidationConfig(TableConfig tableConfig) {
+SegmentsValidationAndRetentionConfig validationConfig =
tableConfig.getValidationConfig();
+if (validationConfig != null) {
+ String peerSegmentDownloadScheme =
validationConfig.getPeerSegmentDownloadScheme();
+ if (peerSegmentDownloadScheme != null) {
+if (!"http".equalsIgnoreCase(peerSegmentDownloadScheme) &&
!"https".equalsIgnoreCase(peerSegmentDownloadScheme)) {
+ throw new IllegalStateException("Invalid value '" +
peerSegmentDownloadScheme + "' for peerSegmentDownloadScheme. Must be one of
http nor https" );
+}
+ }
+}
+ }
}
diff --git
a/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
b/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
index 4d34012..d2d2ebf 100644
---
a/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
+++
b/pinot-common/src/test/java/org/apache/pinot/common/utils/config/TableConfigSerDeTest.java
@@ -243,6 +243,17 @@ public class TableConfigSerDeTest {
checkTableConfigWithUpsertConfig(JsonUtils.stringToObject(tableConfig.toJsonString(),
TableConfig.class));
checkTableConfigWithUpsertConfig(TableConfigUtils.fromZNRecord(TableConfigUtils.toZNRecord(tableConfig)));
}
+{
+ // with SegmentsValidationAndRetentionConfig
+ TableConfig tableConfig =
tableConfigBuilder.setPeerSegmentDownloadScheme("http").build();
+
checkSegmentsValidationAndRetentionConfig(JsonUtils.stringToObject(tableConfig.toJsonString(),
TableConfig.class));
+
checkSegmentsValidationAndRetentionConfig(TableConfigUtils.fromZNRecord(TableConfigUtils.toZNRecord(tableConfig)));
+}
+ }
+
+ private void checkSegmentsValidationAndRetentionConfig(TableConfig
tableConfig) {
+// TODO validate other fields of SegmentsValidationAndRetentionConfig.
+
assertEquals(tableConfig.getValidationConfig().getPeerSegmentDownloadScheme(),
"http");
}
private void checkDefaultTableConfig(TableConfig tableConfig) {
diff --git
a/pinot-spi/src/main/java/org/apache/pinot/spi/config/table/SegmentsValidationAndRetentionConfig.java
b/pinot-spi/src/main/java/org/apache/pinot/spi/config/table/SegmentsValidationAndRetentionConfig.java
index 4e0f3e2..30451ab 100644
---
a/pinot-spi/src/main/java/org/apache/pinot/spi/config/table/Segme
[incubator-pinot] branch master updated (cee2b23 -> d8ea89c)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from cee2b23 [TE] add mock events loader to pinot-thirdeye docker defaults (#5454) add d8ea89c Create PULL_REQUEST_TEMPLATE.md (#5345) No new revisions were added by this update. Summary of changes: .github/PULL_REQUEST_TEMPLATE.md | 28 .gitignore | 4 +++- 2 files changed, 31 insertions(+), 1 deletion(-) create mode 100644 .github/PULL_REQUEST_TEMPLATE.md - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adding-pr-template updated (ebfad8f -> 7b2b0a8)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch adding-pr-template in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from ebfad8f Update PULL_REQUEST_TEMPLATE.md add 7b2b0a8 Update PULL_REQUEST_TEMPLATE.md No new revisions were added by this update. Summary of changes: .github/PULL_REQUEST_TEMPLATE.md | 4 1 file changed, 4 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Set hashmap initial size to 16 in DictionaryBasedGroupKeyGenerator. (#5421)
This is an automated email from the ASF dual-hosted git repository.
mcvsubbu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 4f01114 Set hashmap initial size to 16 in
DictionaryBasedGroupKeyGenerator. (#5421)
4f01114 is described below
commit 4f01114dd08de7d2f23d001ab2253328550d8895
Author: Xiang Fu
AuthorDate: Wed May 20 17:38:32 2020 -0700
Set hashmap initial size to 16 in DictionaryBasedGroupKeyGenerator. (#5421)
---
.../aggregation/groupby/DictionaryBasedGroupKeyGenerator.java | 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGenerator.java
b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGenerator.java
index e2da862..c163ffd 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGenerator.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/groupby/DictionaryBasedGroupKeyGenerator.java
@@ -61,6 +61,7 @@ import org.apache.pinot.core.segment.index.readers.Dictionary;
* bounded by the number of groups limit (globalGroupIdUpperBound is always
smaller or equal to numGroupsLimit).
*/
public class DictionaryBasedGroupKeyGenerator implements GroupKeyGenerator {
+ private final static int DEFAULT_HASH_MAP_INITIAL_SIZE = 16;
private final TransformExpressionTree[] _groupByExpressions;
private final int _numGroupByExpressions;
private final int[] _cardinalities;
@@ -109,15 +110,15 @@ public class DictionaryBasedGroupKeyGenerator implements
GroupKeyGenerator {
if (longOverflow) {
_globalGroupIdUpperBound = numGroupsLimit;
- _rawKeyHolder = new ArrayMapBasedHolder(_globalGroupIdUpperBound);
+ _rawKeyHolder = new ArrayMapBasedHolder(DEFAULT_HASH_MAP_INITIAL_SIZE);
} else {
if (cardinalityProduct > Integer.MAX_VALUE) {
_globalGroupIdUpperBound = numGroupsLimit;
-_rawKeyHolder = new LongMapBasedHolder(_globalGroupIdUpperBound);
+_rawKeyHolder = new LongMapBasedHolder(DEFAULT_HASH_MAP_INITIAL_SIZE);
} else {
_globalGroupIdUpperBound = Math.min((int) cardinalityProduct,
numGroupsLimit);
if (cardinalityProduct > arrayBasedThreshold) {
- _rawKeyHolder = new IntMapBasedHolder(_globalGroupIdUpperBound);
+ _rawKeyHolder = new IntMapBasedHolder(DEFAULT_HASH_MAP_INITIAL_SIZE);
} else {
_rawKeyHolder = new ArrayBasedHolder();
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Upgrade helix to 0.9.7 (#5411)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 1e0cc16 Upgrade helix to 0.9.7 (#5411) 1e0cc16 is described below commit 1e0cc16258257f5f1084f8e956a33f93c27b0d70 Author: Subbu Subramaniam AuthorDate: Tue May 19 16:12:04 2020 -0700 Upgrade helix to 0.9.7 (#5411) Helix release notes: http://helix.apache.org/0.9.7-docs/releasenotes/release-0.9.7.html --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index b9c961f..2b6ae90 100644 --- a/pom.xml +++ b/pom.xml @@ -124,7 +124,7 @@ 1.7.6 1.8.0 -0.9.4 +0.9.7 0.7 2.9.8 1.9.21 - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adding-pr-template updated (444d9fd -> ebfad8f)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch adding-pr-template in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 444d9fd Update PULL_REQUEST_TEMPLATE.md add ebfad8f Update PULL_REQUEST_TEMPLATE.md No new revisions were added by this update. Summary of changes: .github/PULL_REQUEST_TEMPLATE.md | 8 1 file changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adding-pr-template updated (153f29b -> 444d9fd)
This is an automated email from the ASF dual-hosted git repository. mcvsubbu pushed a change to branch adding-pr-template in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 153f29b Update PULL_REQUEST_TEMPLATE.md add 444d9fd Update PULL_REQUEST_TEMPLATE.md No new revisions were added by this update. Summary of changes: .github/PULL_REQUEST_TEMPLATE.md | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
