[incubator-pinot] branch master updated (5503d4d -> f15a7b0)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 5503d4d Rewrite single value IN predicate as EQUALITY. (#4847) add f15a7b0 Add error for empty instance list for tenant API (#4969) No new revisions were added by this update. Summary of changes: .../api/resources/PinotTenantRestletResource.java | 38 +- .../controller/helix/ControllerTenantTest.java | 10 ++ 2 files changed, 39 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch tenantNotFound updated (8f77898 -> 4589677)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch tenantNotFound in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 8f77898 Optimizing add 4589677 Optimizing This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (8f77898) \ N -- N -- N refs/heads/tenantNotFound (4589677) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../controller/api/resources/PinotTenantRestletResource.java | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch tenantNotFound updated (25bfe80 -> 0b5592b)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch tenantNotFound in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 25bfe80 Add error for empty instance list for tenant API discard a5fd827 Tenant not found new 0b5592b Add error for empty instance list for tenant API This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (25bfe80) \ N -- N -- N refs/heads/tenantNotFound (0b5592b) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Tenant not found
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch tenantNotFound
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit a5fd827af7f0205e75d4c7e0006c80f3738b732a
Author: Jennifer Dai
AuthorDate: Thu Jan 9 15:06:17 2020 -0800
Tenant not found
---
.../api/resources/PinotTenantRestletResource.java| 20 +++-
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTenantRestletResource.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTenantRestletResource.java
index 5f11c52..75861fd 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTenantRestletResource.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotTenantRestletResource.java
@@ -46,10 +46,10 @@ import org.apache.pinot.common.config.TableConfig;
import org.apache.pinot.common.config.Tenant;
import org.apache.pinot.common.metrics.ControllerMeter;
import org.apache.pinot.common.metrics.ControllerMetrics;
-import org.apache.pinot.spi.utils.JsonUtils;
import org.apache.pinot.common.utils.TenantRole;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
import org.apache.pinot.controller.helix.core.PinotResourceManagerResponse;
+import org.apache.pinot.spi.utils.JsonUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -259,19 +259,21 @@ public class PinotTenantRestletResource {
private String listInstancesForTenant(String tenantName, String tenantType) {
ObjectNode resourceGetRet = JsonUtils.newObjectNode();
+Set allServerInstances =
pinotHelixResourceManager.getAllInstancesForServerTenant(tenantName);
+Set allBrokerInstances =
pinotHelixResourceManager.getAllInstancesForBrokerTenant(tenantName);
+if (allServerInstances.isEmpty() && allBrokerInstances.isEmpty()) {
+ throw new ControllerApplicationException(LOGGER, "Failed to find any
instances for tenant: " + tenantName,
+ Response.Status.NOT_FOUND);
+}
if (tenantType == null) {
- resourceGetRet.set("ServerInstances",
-
JsonUtils.objectToJsonNode(pinotHelixResourceManager.getAllInstancesForServerTenant(tenantName)));
- resourceGetRet.set("BrokerInstances",
-
JsonUtils.objectToJsonNode(pinotHelixResourceManager.getAllInstancesForBrokerTenant(tenantName)));
+ resourceGetRet.set("ServerInstances",
JsonUtils.objectToJsonNode(allServerInstances));
+ resourceGetRet.set("BrokerInstances",
JsonUtils.objectToJsonNode(allBrokerInstances));
} else {
if (tenantType.equalsIgnoreCase("server")) {
-resourceGetRet.set("ServerInstances",
-
JsonUtils.objectToJsonNode(pinotHelixResourceManager.getAllInstancesForServerTenant(tenantName)));
+resourceGetRet.set("ServerInstances",
JsonUtils.objectToJsonNode(allServerInstances));
}
if (tenantType.equalsIgnoreCase("broker")) {
-resourceGetRet.set("BrokerInstances",
-
JsonUtils.objectToJsonNode(pinotHelixResourceManager.getAllInstancesForBrokerTenant(tenantName)));
+resourceGetRet.set("BrokerInstances",
JsonUtils.objectToJsonNode(allBrokerInstances));
}
}
resourceGetRet.put(TENANT_NAME, tenantName);
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch tenantNotFound created (now a5fd827)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch tenantNotFound in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at a5fd827 Tenant not found This branch includes the following new commits: new a5fd827 Tenant not found The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (78bd737 -> 08c0c2b)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 78bd737 Clean up the un-necessary split for IN/NOT_IN predicates (#4924) add 08c0c2b Modifying segment fetcher doc to v2 endpoint (#4923) No new revisions were added by this update. Summary of changes: docs/segment_fetcher.rst | 26 ++ 1 file changed, 26 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch doc updated (37308e0 -> 646bc12)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch doc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 37308e0 Adding crypter configs and additional info about v1 endpoint add 646bc12 Adding crypter configs and additional info about v1 endpoint This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (37308e0) \ N -- N -- N refs/heads/doc (646bc12) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: docs/segment_fetcher.rst | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch doc updated (35fa66c -> 37308e0)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch doc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 35fa66c Adding crypter configs and additional info about v1 endpoint add 37308e0 Adding crypter configs and additional info about v1 endpoint This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (35fa66c) \ N -- N -- N refs/heads/doc (37308e0) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: docs/segment_fetcher.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch doc updated (56a09ba -> 35fa66c)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch doc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 56a09ba Modifying segment fetcher doc to v2 endpoint add 35fa66c Adding crypter configs and additional info about v1 endpoint No new revisions were added by this update. Summary of changes: docs/segment_fetcher.rst | 23 +++ 1 file changed, 23 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch doc created (now 56a09ba)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch doc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 56a09ba Modifying segment fetcher doc to v2 endpoint This branch includes the following new commits: new 56a09ba Modifying segment fetcher doc to v2 endpoint The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Modifying segment fetcher doc to v2 endpoint
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a commit to branch doc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 56a09bae4a7e988b3351c0e0dba41587d120b41b Author: Jennifer Dai AuthorDate: Thu Dec 12 15:07:44 2019 -0800 Modifying segment fetcher doc to v2 endpoint --- docs/segment_fetcher.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/segment_fetcher.rst b/docs/segment_fetcher.rst index 625cb16..0752631 100644 --- a/docs/segment_fetcher.rst +++ b/docs/segment_fetcher.rst @@ -77,7 +77,7 @@ For example, the following curl requests to Controller will notify it to downloa .. code-block:: none - curl -X POST -H "UPLOAD_TYPE:URI" -H "DOWNLOAD_URI:hdfs://nameservice1/hadoop/path/to/segment/file.gz" -H "content-type:application/json" -d '' localhost:9000/segments + curl -X POST -H "UPLOAD_TYPE:URI" -H "DOWNLOAD_URI:hdfs://nameservice1/hadoop/path/to/segment/file.gz" -H "content-type:application/json" -d '' localhost:9000/v2/segments Implement your own segment fetcher for other systems - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (23d1437 -> ef42fc4)
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from 23d1437 Adding example of k8s presto-worker deployment (#4895)
add ef42fc4 Moving crypter classes to pinot-spi (#4897)
No new revisions were added by this update.
Summary of changes:
.../main/java/org/apache/pinot/controller/ControllerStarter.java| 2 +-
.../api/resources/PinotSegmentUploadDownloadRestletResource.java| 6 +++---
.../apache/pinot/{core => spi}/crypt/PinotCrypterFactoryTest.java | 2 +-
.../src/main/java/org/apache/pinot/minion/MinionStarter.java| 2 +-
.../apache/pinot/server/starter/helix/SegmentFetcherAndLoader.java | 5 ++---
.../src/main/java/org/apache/pinot/spi}/crypt/NoOpPinotCrypter.java | 2 +-
.../src/main/java/org/apache/pinot/spi}/crypt/PinotCrypter.java | 2 +-
.../main/java/org/apache/pinot/spi}/crypt/PinotCrypterFactory.java | 2 +-
8 files changed, 11 insertions(+), 12 deletions(-)
rename pinot-core/src/test/java/org/apache/pinot/{core =>
spi}/crypt/PinotCrypterFactoryTest.java (98%)
rename {pinot-core/src/main/java/org/apache/pinot/core =>
pinot-spi/src/main/java/org/apache/pinot/spi}/crypt/NoOpPinotCrypter.java (97%)
rename {pinot-core/src/main/java/org/apache/pinot/core =>
pinot-spi/src/main/java/org/apache/pinot/spi}/crypt/PinotCrypter.java (97%)
rename {pinot-core/src/main/java/org/apache/pinot/core =>
pinot-spi/src/main/java/org/apache/pinot/spi}/crypt/PinotCrypterFactory.java
(98%)
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Moving crypter classes to pinot-spi
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch spicrypt
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 0ee28402d38ecc29c5bb3b1ddd217a7eb675daba
Author: Jennifer Dai
AuthorDate: Fri Dec 6 07:42:49 2019 -0800
Moving crypter classes to pinot-spi
---
.../main/java/org/apache/pinot/controller/ControllerStarter.java| 2 +-
.../api/resources/PinotSegmentUploadDownloadRestletResource.java| 6 +++---
.../apache/pinot/{core => spi}/crypt/PinotCrypterFactoryTest.java | 2 +-
.../src/main/java/org/apache/pinot/minion/MinionStarter.java| 2 +-
.../apache/pinot/server/starter/helix/SegmentFetcherAndLoader.java | 5 ++---
.../src/main/java/org/apache/pinot/spi}/crypt/NoOpPinotCrypter.java | 2 +-
.../src/main/java/org/apache/pinot/spi}/crypt/PinotCrypter.java | 2 +-
.../main/java/org/apache/pinot/spi}/crypt/PinotCrypterFactory.java | 2 +-
8 files changed, 11 insertions(+), 12 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java
index e92d2f3..984953a 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerStarter.java
@@ -71,7 +71,7 @@ import org.apache.pinot.controller.helix.starter.HelixConfig;
import org.apache.pinot.controller.validation.BrokerResourceValidationManager;
import org.apache.pinot.controller.validation.OfflineSegmentIntervalChecker;
import org.apache.pinot.controller.validation.RealtimeSegmentValidationManager;
-import org.apache.pinot.core.crypt.PinotCrypterFactory;
+import org.apache.pinot.spi.crypt.PinotCrypterFactory;
import org.apache.pinot.core.periodictask.PeriodicTask;
import org.apache.pinot.core.periodictask.PeriodicTaskScheduler;
import org.apache.pinot.filesystem.PinotFSFactory;
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
index b7574bf..ca77291 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
@@ -70,9 +70,9 @@ import
org.apache.pinot.controller.api.access.AccessControlFactory;
import org.apache.pinot.controller.api.upload.SegmentValidator;
import org.apache.pinot.controller.api.upload.ZKOperator;
import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
-import org.apache.pinot.core.crypt.NoOpPinotCrypter;
-import org.apache.pinot.core.crypt.PinotCrypter;
-import org.apache.pinot.core.crypt.PinotCrypterFactory;
+import org.apache.pinot.spi.crypt.NoOpPinotCrypter;
+import org.apache.pinot.spi.crypt.PinotCrypter;
+import org.apache.pinot.spi.crypt.PinotCrypterFactory;
import org.apache.pinot.core.metadata.DefaultMetadataExtractor;
import org.apache.pinot.core.metadata.MetadataExtractorFactory;
import org.apache.pinot.spi.filesystem.PinotFS;
diff --git
a/pinot-core/src/test/java/org/apache/pinot/core/crypt/PinotCrypterFactoryTest.java
b/pinot-core/src/test/java/org/apache/pinot/spi/crypt/PinotCrypterFactoryTest.java
similarity index 98%
rename from
pinot-core/src/test/java/org/apache/pinot/core/crypt/PinotCrypterFactoryTest.java
rename to
pinot-core/src/test/java/org/apache/pinot/spi/crypt/PinotCrypterFactoryTest.java
index 15c19ef..398558c 100644
---
a/pinot-core/src/test/java/org/apache/pinot/core/crypt/PinotCrypterFactoryTest.java
+++
b/pinot-core/src/test/java/org/apache/pinot/spi/crypt/PinotCrypterFactoryTest.java
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
-package org.apache.pinot.core.crypt;
+package org.apache.pinot.spi.crypt;
import java.io.File;
import org.apache.commons.configuration.Configuration;
diff --git
a/pinot-minion/src/main/java/org/apache/pinot/minion/MinionStarter.java
b/pinot-minion/src/main/java/org/apache/pinot/minion/MinionStarter.java
index f6bc0b1..537d299 100644
--- a/pinot-minion/src/main/java/org/apache/pinot/minion/MinionStarter.java
+++ b/pinot-minion/src/main/java/org/apache/pinot/minion/MinionStarter.java
@@ -39,7 +39,7 @@ import
org.apache.pinot.common.utils.ClientSSLContextGenerator;
import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.NetUtil;
import org.apache.pinot.common.utils.ServiceStatus;
-import org.apache.pinot.core.crypt.PinotCrypterFactory;
+import org.apache.pinot.spi.crypt.PinotCrypterFactory;
import org.apache.pinot.filesystem.PinotFSFactory;
imp
[incubator-pinot] branch spicrypt created (now 0ee2840)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch spicrypt in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 0ee2840 Moving crypter classes to pinot-spi This branch includes the following new commits: new 0ee2840 Moving crypter classes to pinot-spi The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch trypt updated (4419a3b -> 3f118c2)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch trypt in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 4419a3b Add targz to SegmentIndexCreationImpl new 3f118c2 Add targz to SegmentIndexCreationImpl This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (4419a3b) \ N -- N -- N refs/heads/trypt (3f118c2) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/pinot/hadoop/job/mappers/SegmentCreationMapper.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Add targz to SegmentIndexCreationImpl
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch trypt
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 3f118c2a3d826ae5568156d8f97f0d64e7d913c8
Author: Jennifer Dai
AuthorDate: Tue Dec 3 14:57:41 2019 -0800
Add targz to SegmentIndexCreationImpl
---
.../generator/SegmentGeneratorConfig.java | 10
.../impl/SegmentIndexCreationDriverImpl.java | 20 +---
.../hadoop/job/mappers/SegmentCreationMapper.java | 27 --
.../pinot/ingestion/common/JobConfigConstants.java | 3 +++
...mentBuildPushOfflineClusterIntegrationTest.java | 4 +++-
5 files changed, 38 insertions(+), 26 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
index 1d42001..4e57c6f 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
@@ -114,6 +114,7 @@ public class SegmentGeneratorConfig {
private boolean _onHeap = false;
private boolean _skipTimeValueCheck = false;
private boolean _nullHandlingEnabled = false;
+ private boolean _createTarGz = false;
public SegmentGeneratorConfig() {
}
@@ -164,6 +165,7 @@ public class SegmentGeneratorConfig {
_recordReaderPath = config._recordReaderPath;
_skipTimeValueCheck = config._skipTimeValueCheck;
_nullHandlingEnabled = config._nullHandlingEnabled;
+_createTarGz = config._createTarGz;
}
/**
@@ -352,6 +354,14 @@ public class SegmentGeneratorConfig {
_recordReaderPath = recordReaderPath;
}
+ public boolean isCreateTarGz() {
+return _createTarGz;
+ }
+
+ public void setCreateTarGz(boolean createTarGz) {
+_createTarGz = createTarGz;
+ }
+
public String getOutDir() {
return _outDir;
}
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
index 3941f66..1fdaa7e 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
@@ -32,12 +32,8 @@ import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import org.apache.commons.io.FileUtils;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.MetricFieldSpec;
-import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.common.data.StarTreeIndexSpec;
-import org.apache.pinot.spi.data.readers.GenericRow;
-import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.common.utils.TarGzCompressionUtils;
import org.apache.pinot.core.data.readers.RecordReaderFactory;
import org.apache.pinot.core.data.recordtransformer.CompositeTransformer;
import org.apache.pinot.core.data.recordtransformer.RecordTransformer;
@@ -65,6 +61,11 @@ import org.apache.pinot.core.startree.hll.HllUtil;
import org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder;
import org.apache.pinot.core.startree.v2.builder.StarTreeV2BuilderConfig;
import org.apache.pinot.core.util.CrcUtils;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.MetricFieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
import org.apache.pinot.startree.hll.HllConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -388,9 +389,6 @@ public class SegmentIndexCreationDriverImpl implements
SegmentIndexCreationDrive
// Move the temporary directory into its final location
FileUtils.moveDirectory(tempIndexDir, segmentOutputDir);
-// Delete the temporary directory
-FileUtils.deleteQuietly(tempIndexDir);
-
// Convert segment format if necessary
convertFormatIfNeeded(segmentOutputDir);
@@ -415,6 +413,12 @@ public class SegmentIndexCreationDriverImpl implements
SegmentIndexCreationDrive
// Persist creation metadata to disk
persistCreationMeta(segmentOutputDir, crc, creationTime);
+if (config.isCreateTarGz()) {
+ File tarGzPath = new File(outputDir, segmentName +
TarGzCompressionUtils.TAR_GZ_FILE_EXTENSION);
+
TarGzCompressionUtils.createTarGzOfDirectory(segmentOutputDir.getAbsolutePath(),
tarGzPath.getAbsolutePath());
+ FileUtils.deleteQuietly(segmentOutputDir);
+}
+
LOGGER.info("Driver, record read time : {}", totalRecordReadTime);
LOGGER.info("Driver, stats collector time : {}", totalStatsCollectorTime);
LOGGER.in
[incubator-pinot] 01/01: Add targz to SegmentIndexCreationImpl
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch trypt
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 4419a3b86c19b4abc5127161eb3797cf57a66496
Author: Jennifer Dai
AuthorDate: Tue Dec 3 14:57:41 2019 -0800
Add targz to SegmentIndexCreationImpl
---
.../generator/SegmentGeneratorConfig.java | 10
.../impl/SegmentIndexCreationDriverImpl.java | 20 +---
.../hadoop/job/mappers/SegmentCreationMapper.java | 27 --
.../pinot/ingestion/common/JobConfigConstants.java | 3 +++
...mentBuildPushOfflineClusterIntegrationTest.java | 4 +++-
5 files changed, 38 insertions(+), 26 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
index 1d42001..4e57c6f 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
@@ -114,6 +114,7 @@ public class SegmentGeneratorConfig {
private boolean _onHeap = false;
private boolean _skipTimeValueCheck = false;
private boolean _nullHandlingEnabled = false;
+ private boolean _createTarGz = false;
public SegmentGeneratorConfig() {
}
@@ -164,6 +165,7 @@ public class SegmentGeneratorConfig {
_recordReaderPath = config._recordReaderPath;
_skipTimeValueCheck = config._skipTimeValueCheck;
_nullHandlingEnabled = config._nullHandlingEnabled;
+_createTarGz = config._createTarGz;
}
/**
@@ -352,6 +354,14 @@ public class SegmentGeneratorConfig {
_recordReaderPath = recordReaderPath;
}
+ public boolean isCreateTarGz() {
+return _createTarGz;
+ }
+
+ public void setCreateTarGz(boolean createTarGz) {
+_createTarGz = createTarGz;
+ }
+
public String getOutDir() {
return _outDir;
}
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
index 3941f66..1fdaa7e 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
@@ -32,12 +32,8 @@ import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import org.apache.commons.io.FileUtils;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.MetricFieldSpec;
-import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.common.data.StarTreeIndexSpec;
-import org.apache.pinot.spi.data.readers.GenericRow;
-import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.common.utils.TarGzCompressionUtils;
import org.apache.pinot.core.data.readers.RecordReaderFactory;
import org.apache.pinot.core.data.recordtransformer.CompositeTransformer;
import org.apache.pinot.core.data.recordtransformer.RecordTransformer;
@@ -65,6 +61,11 @@ import org.apache.pinot.core.startree.hll.HllUtil;
import org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder;
import org.apache.pinot.core.startree.v2.builder.StarTreeV2BuilderConfig;
import org.apache.pinot.core.util.CrcUtils;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.MetricFieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
import org.apache.pinot.startree.hll.HllConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -388,9 +389,6 @@ public class SegmentIndexCreationDriverImpl implements
SegmentIndexCreationDrive
// Move the temporary directory into its final location
FileUtils.moveDirectory(tempIndexDir, segmentOutputDir);
-// Delete the temporary directory
-FileUtils.deleteQuietly(tempIndexDir);
-
// Convert segment format if necessary
convertFormatIfNeeded(segmentOutputDir);
@@ -415,6 +413,12 @@ public class SegmentIndexCreationDriverImpl implements
SegmentIndexCreationDrive
// Persist creation metadata to disk
persistCreationMeta(segmentOutputDir, crc, creationTime);
+if (config.isCreateTarGz()) {
+ File tarGzPath = new File(outputDir, segmentName +
TarGzCompressionUtils.TAR_GZ_FILE_EXTENSION);
+
TarGzCompressionUtils.createTarGzOfDirectory(segmentOutputDir.getAbsolutePath(),
tarGzPath.getAbsolutePath());
+ FileUtils.deleteQuietly(segmentOutputDir);
+}
+
LOGGER.info("Driver, record read time : {}", totalRecordReadTime);
LOGGER.info("Driver, stats collector time : {}", totalStatsCollectorTime);
LOGGER.in
[incubator-pinot] branch trypt updated (05247b0 -> 4419a3b)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch trypt in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 05247b0 Adding tar gz compression during segment creation new 4419a3b Add targz to SegmentIndexCreationImpl This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (05247b0) \ N -- N -- N refs/heads/trypt (4419a3b) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/pinot/hadoop/job/mappers/SegmentCreationMapper.java| 8 ++-- 1 file changed, 2 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Adding tar gz compression during segment creation
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch trypt
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 05247b0257d8435f96d4dbc948bad7650ce2da31
Author: Jennifer Dai
AuthorDate: Tue Dec 3 14:43:36 2019 -0800
Adding tar gz compression during segment creation
---
.../generator/SegmentGeneratorConfig.java| 10 ++
.../creator/impl/SegmentIndexCreationDriverImpl.java | 20
.../hadoop/job/mappers/SegmentCreationMapper.java| 19 ---
.../pinot/ingestion/common/JobConfigConstants.java | 3 +++
...egmentBuildPushOfflineClusterIntegrationTest.java | 4 +++-
5 files changed, 36 insertions(+), 20 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
index 1d42001..4e57c6f 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
@@ -114,6 +114,7 @@ public class SegmentGeneratorConfig {
private boolean _onHeap = false;
private boolean _skipTimeValueCheck = false;
private boolean _nullHandlingEnabled = false;
+ private boolean _createTarGz = false;
public SegmentGeneratorConfig() {
}
@@ -164,6 +165,7 @@ public class SegmentGeneratorConfig {
_recordReaderPath = config._recordReaderPath;
_skipTimeValueCheck = config._skipTimeValueCheck;
_nullHandlingEnabled = config._nullHandlingEnabled;
+_createTarGz = config._createTarGz;
}
/**
@@ -352,6 +354,14 @@ public class SegmentGeneratorConfig {
_recordReaderPath = recordReaderPath;
}
+ public boolean isCreateTarGz() {
+return _createTarGz;
+ }
+
+ public void setCreateTarGz(boolean createTarGz) {
+_createTarGz = createTarGz;
+ }
+
public String getOutDir() {
return _outDir;
}
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
index 3941f66..1fdaa7e 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentIndexCreationDriverImpl.java
@@ -32,12 +32,8 @@ import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import org.apache.commons.io.FileUtils;
-import org.apache.pinot.spi.data.FieldSpec;
-import org.apache.pinot.spi.data.MetricFieldSpec;
-import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.common.data.StarTreeIndexSpec;
-import org.apache.pinot.spi.data.readers.GenericRow;
-import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.common.utils.TarGzCompressionUtils;
import org.apache.pinot.core.data.readers.RecordReaderFactory;
import org.apache.pinot.core.data.recordtransformer.CompositeTransformer;
import org.apache.pinot.core.data.recordtransformer.RecordTransformer;
@@ -65,6 +61,11 @@ import org.apache.pinot.core.startree.hll.HllUtil;
import org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder;
import org.apache.pinot.core.startree.v2.builder.StarTreeV2BuilderConfig;
import org.apache.pinot.core.util.CrcUtils;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.MetricFieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
import org.apache.pinot.startree.hll.HllConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -388,9 +389,6 @@ public class SegmentIndexCreationDriverImpl implements
SegmentIndexCreationDrive
// Move the temporary directory into its final location
FileUtils.moveDirectory(tempIndexDir, segmentOutputDir);
-// Delete the temporary directory
-FileUtils.deleteQuietly(tempIndexDir);
-
// Convert segment format if necessary
convertFormatIfNeeded(segmentOutputDir);
@@ -415,6 +413,12 @@ public class SegmentIndexCreationDriverImpl implements
SegmentIndexCreationDrive
// Persist creation metadata to disk
persistCreationMeta(segmentOutputDir, crc, creationTime);
+if (config.isCreateTarGz()) {
+ File tarGzPath = new File(outputDir, segmentName +
TarGzCompressionUtils.TAR_GZ_FILE_EXTENSION);
+
TarGzCompressionUtils.createTarGzOfDirectory(segmentOutputDir.getAbsolutePath(),
tarGzPath.getAbsolutePath());
+ FileUtils.deleteQuietly(segmentOutputDir);
+}
+
LOGGER.info("Driver, record read time : {}", totalRecordReadTime);
LOGGER.info("Driver, stats collector time : {}", tot
[incubator-pinot] branch trypt created (now 05247b0)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch trypt in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 05247b0 Adding tar gz compression during segment creation This branch includes the following new commits: new 05247b0 Adding tar gz compression during segment creation The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch hencrypt updated (c8dd2f7 -> 0afbff1)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch hencrypt in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard c8dd2f7 Encrypt in hadoop job new 0afbff1 Encrypt in hadoop job This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (c8dd2f7) \ N -- N -- N refs/heads/hencrypt (0afbff1) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Encrypt in hadoop job
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch hencrypt
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 0afbff1790627b52df8b000b556560245febcc0f
Author: Jennifer Dai
AuthorDate: Mon Dec 2 12:58:28 2019 -0800
Encrypt in hadoop job
---
.../org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java | 11 +++
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git
a/pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
b/pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
index 297807a..d01e393 100644
---
a/pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
+++
b/pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
@@ -38,7 +38,6 @@ import
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.pinot.common.config.SegmentsValidationAndRetentionConfig;
import org.apache.pinot.common.config.TableConfig;
-import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.common.utils.StringUtil;
import org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper;
import org.apache.pinot.hadoop.utils.PinotHadoopJobPreparationHelper;
@@ -132,13 +131,17 @@ public class HadoopSegmentCreationJob extends
SegmentCreationJob {
throw new RuntimeException("Job failed: " + job);
}
-moveSegmentsToOutputDir();
+encryptSegmentsAndMove();
// Delete the staging directory
_logger.info("Deleting the staging directory: {}", _stagingDir);
_outputDirFileSystem.delete(new Path(_stagingDir), true);
}
+ public void encryptSegmentsAndMove() throws IOException {
+moveSegmentsToOutputDir(JobConfigConstants.SEGMENT_TAR_DIR);
+ }
+
protected void validateTableConfig(TableConfig tableConfig) {
SegmentsValidationAndRetentionConfig validationConfig =
tableConfig.getValidationConfig();
@@ -171,9 +174,9 @@ public class HadoopSegmentCreationJob extends
SegmentCreationJob {
protected void addAdditionalJobProperties(Job job) {
}
- protected void moveSegmentsToOutputDir()
+ protected void moveSegmentsToOutputDir(String simpleDirName)
throws IOException {
-Path segmentTarDir = new Path(new Path(_stagingDir, "output"),
JobConfigConstants.SEGMENT_TAR_DIR);
+Path segmentTarDir = new Path(new Path(_stagingDir, "output"),
simpleDirName);
movePath(_outputDirFileSystem, segmentTarDir.toString(), _outputDir, true);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Encrypt in hadoop job
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch hencrypt
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit c8dd2f730ebc082e7b3b084e32b7b6dc15458730
Author: Jennifer Dai
AuthorDate: Mon Dec 2 12:58:28 2019 -0800
Encrypt in hadoop job
---
.../org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java | 11 +++
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git
a/pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
b/pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
index 297807a..94e0b96 100644
---
a/pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
+++
b/pinot-ingestion-jobs/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/HadoopSegmentCreationJob.java
@@ -38,7 +38,6 @@ import
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.pinot.common.config.SegmentsValidationAndRetentionConfig;
import org.apache.pinot.common.config.TableConfig;
-import org.apache.pinot.spi.data.Schema;
import org.apache.pinot.common.utils.StringUtil;
import org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper;
import org.apache.pinot.hadoop.utils.PinotHadoopJobPreparationHelper;
@@ -132,13 +131,17 @@ public class HadoopSegmentCreationJob extends
SegmentCreationJob {
throw new RuntimeException("Job failed: " + job);
}
-moveSegmentsToOutputDir();
+encryptSegments();
// Delete the staging directory
_logger.info("Deleting the staging directory: {}", _stagingDir);
_outputDirFileSystem.delete(new Path(_stagingDir), true);
}
+ public void encryptSegments() throws IOException {
+moveSegmentsToOutputDir(JobConfigConstants.SEGMENT_TAR_DIR);
+ }
+
protected void validateTableConfig(TableConfig tableConfig) {
SegmentsValidationAndRetentionConfig validationConfig =
tableConfig.getValidationConfig();
@@ -171,9 +174,9 @@ public class HadoopSegmentCreationJob extends
SegmentCreationJob {
protected void addAdditionalJobProperties(Job job) {
}
- protected void moveSegmentsToOutputDir()
+ protected void moveSegmentsToOutputDir(String simpleDirName)
throws IOException {
-Path segmentTarDir = new Path(new Path(_stagingDir, "output"),
JobConfigConstants.SEGMENT_TAR_DIR);
+Path segmentTarDir = new Path(new Path(_stagingDir, "output"),
simpleDirName);
movePath(_outputDirFileSystem, segmentTarDir.toString(), _outputDir, true);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch hencrypt created (now c8dd2f7)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch hencrypt in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at c8dd2f7 Encrypt in hadoop job This branch includes the following new commits: new c8dd2f7 Encrypt in hadoop job The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (98f8c9a -> c0f8b7a)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 98f8c9a [TE] Update AI Availability dashboard to v2 (#4857) add c0f8b7a Fix the controller rest API for non local FS. (#4808) No new revisions were added by this update. Summary of changes: .../PinotSegmentUploadDownloadRestletResource.java | 51 ++ 1 file changed, 42 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (2f18f65 -> 0f64e4a)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 2f18f65 [TE] add detection health for the alerts list (#4842) add 0f64e4a Fixing encryption for .tar.gz upload (#4855) No new revisions were added by this update. Summary of changes: .../api/resources/PinotSegmentUploadDownloadRestletResource.java| 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Fixing encryption for .tar.gz upload
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch crypt
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 7901df7b54264267f6cc6c7e914067b83d4dcfb4
Author: Jennifer Dai
AuthorDate: Mon Nov 25 10:41:45 2019 -0800
Fixing encryption for .tar.gz upload
---
.../api/resources/PinotSegmentUploadDownloadRestletResource.java| 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
index 22a7dea..6d36137 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
@@ -192,7 +192,7 @@ public class PinotSegmentUploadDownloadRestletResource {
metadataProviderClass);
break;
case SEGMENT:
- getFileFromMultipart(multiPart, tempDecryptedFile);
+ getFileFromMultipart(multiPart, tempEncryptedFile);
segmentMetadata = getSegmentMetadata(crypterClassName,
tempEncryptedFile, tempDecryptedFile, tempSegmentDir,
metadataProviderClass);
break;
@@ -303,14 +303,14 @@ public class PinotSegmentUploadDownloadRestletResource {
return
MetadataExtractorFactory.create(metadataProviderClass).extractMetadata(tempDecryptedFile,
tempSegmentDir);
}
- private void completeZkOperations(boolean enableParallelPushProtection,
HttpHeaders headers, File tempDecryptedFile,
+ private void completeZkOperations(boolean enableParallelPushProtection,
HttpHeaders headers, File tempEncryptedFile,
FileUploadPathProvider provider, String rawTableName, SegmentMetadata
segmentMetadata, String segmentName,
String zkDownloadURI, boolean moveSegmentToFinalLocation)
throws Exception {
URI finalSegmentLocationURI =
URIUtils.getUri(provider.getBaseDataDirURI().toString(), rawTableName,
URIUtils.encode(segmentName));
ZKOperator zkOperator = new ZKOperator(_pinotHelixResourceManager,
_controllerConf, _controllerMetrics);
-zkOperator.completeSegmentOperations(rawTableName, segmentMetadata,
finalSegmentLocationURI, tempDecryptedFile,
+zkOperator.completeSegmentOperations(rawTableName, segmentMetadata,
finalSegmentLocationURI, tempEncryptedFile,
enableParallelPushProtection, headers, zkDownloadURI,
moveSegmentToFinalLocation);
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch crypt created (now 7901df7)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch crypt in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 7901df7 Fixing encryption for .tar.gz upload This branch includes the following new commits: new 7901df7 Fixing encryption for .tar.gz upload The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (8763494 -> e189f5e)
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from 8763494 Fix documentation on LLC configs (#4839)
add e189f5e Refactoring realtime segment committer (#4713)
No new revisions were added by this update.
Summary of changes:
.../protocols/SegmentCompletionProtocol.java | 13 ++
.../manager/realtime/DefaultSegmentCommitter.java | 53 ++
.../realtime/LLRealtimeSegmentDataManager.java | 99 +++
.../{TimerService.java => SegmentCommitter.java} | 16 +-
.../manager/realtime/SegmentCommitterFactory.java | 49 +
.../manager/realtime/SplitSegmentCommitter.java| 84 +
.../realtime/LLRealtimeSegmentDataManagerTest.java | 6 +-
.../pinot/core/data/readers/PinotSegmentUtil.java | 4 +-
pinot-integration-tests/pom.xml| 4 +
.../DefaultCommitterRealtimeIntegrationTest.java | 197 +
.../tests/LLCRealtimeClusterIntegrationTest.java | 13 +-
11 files changed, 445 insertions(+), 93 deletions(-)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/DefaultSegmentCommitter.java
copy
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/{TimerService.java
=> SegmentCommitter.java} (59%)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SplitSegmentCommitter.java
create mode 100644
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/DefaultCommitterRealtimeIntegrationTest.java
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (d9e5bbd -> 65e2b22)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from d9e5bbd New constructor for params add 65e2b22 Remove visiblefortesting annotation No new revisions were added by this update. Summary of changes: .../test/java/org/apache/pinot/core/data/readers/PinotSegmentUtil.java | 2 -- 1 file changed, 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (4a87848 -> d9e5bbd)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 4a87848 Addressing comments add d9e5bbd New constructor for params No new revisions were added by this update. Summary of changes: .../pinot/common/protocols/SegmentCompletionProtocol.java | 13 + .../core/data/manager/realtime/SegmentCommitterFactory.java | 8 +--- .../core/data/manager/realtime/SplitSegmentCommitter.java | 2 +- 3 files changed, 15 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (8ab0833 -> 4a87848)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 8ab0833 Addressing comments add 4a87848 Addressing comments No new revisions were added by this update. Summary of changes: .../manager/realtime/LLRealtimeSegmentDataManager.java | 16 +++- .../data/manager/realtime/SegmentCommitterFactory.java | 10 -- .../data/manager/realtime/SplitSegmentCommitter.java | 12 +--- .../realtime/LLRealtimeSegmentDataManagerTest.java | 6 +++--- .../tests/DefaultCommitterRealtimeIntegrationTest.java | 2 +- 5 files changed, 24 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (89a1cde -> 8ab0833)
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a change to branch realtimeref
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard 89a1cde Addressing comments
discard 8a63ab3 Fixing doc
discard 8f19edc Addressing comments
discard 5d71622 Dealing with issues from merge
discard 85ef958 Adding test and addressing comments
discard f662d81 Adding committer test
discard 6dfc2bb Addressing comments
discard 30b6e33 Editing test
discard e655d76 Addressing comments
discard be74d25 Fixing test
discard 6e59583 Refactoring realtime segment committer
add ed4e3ae [TE] Initiate docker for ThirdEye (#4780)
add 7790e15 Fix controller download segment api on non-local PinotFS.
(#4757)
add 98140bc Decouple Key from Record (#4798)
add 103f538 [TE] Remove datepartition for presto queries to unblock other
usecases (#4802)
add c9f465a Revert "Fix controller download segment api on non-local
PinotFS. (#4757)" (#4804)
add 64e1342 Small typo fix (confifurations -> configurations) (#4805)
add a92d88d [TE] revert the default settings; update indexes on task
table (#4810)
add 01b1de5 Include dataSchema in the ResultTable to retain column type
information (#4807)
add b9fd128 Adding pinot presto docker image (#4812)
add e92f94b [TE] Fail alert task if unable to notify receipients with
exception of dimensions alerter (#4811)
add 895c7ee Make default timeout to 15s for all queries in tests (#4813)
add e6624fe [TE] Set non additive dataset for Pinot (#4803)
add 36ef2c3 [TE] frontend - harleyjj/homepage - homepage facelift and
default to application tier0-tier1 if none chosen (#4794)
add 5282e11 [TE] frontend - harleyjj/alert-overview - fix report anomaly
bugs (#4816)
add a7b4aff [TE] web - harleyjj/rca - add detectionConfigId to anomaly
entity attributes (#4821)
add 094fffa [TE] frontend - harleyjj/rca - add link from RCA to Alert
Overview whe RCA is exploring an anomaly (#4822)
add 2e221e6 [TE] frontend - harleyjj/preview - disable preview button
instead of removing from DOM (#4485)
add 3ffbab9 [TE][notification] Minor cleaning up of the jira related code
(#4814)
add 1f79d29 Bug Fix: Fix segment download url in SegmentZkMetadata.
(#4825)
add d764ea6 Enhance table rebalance rest API argument check (#4824)
add 648d02a Replace ServerInstance with ServerRoutingInstance for Broker
scatter-gather (#4817)
add eb45b43 [TE] add event driven scheduler (#4777)
add 57f67c9 Refactor segments admin rest APIs (#4806)
add 75dbe8c Enhance RoutingTable with ServerInstance (#4823)
add 687ac69 Reorg k8s related scripts structure and update readme (#4827)
add 85c7f5a Add segment batch deletion rest API (#4828)
add c33da6b [TE] change default time zone to pacific time zone (#4830)
add d493707 Add user guide for pluggable storage for realtime table
(#4783)
add 54a4991 Misc enhancements to the broker routing (#4831)
add 3c00da7 [TE][notification] Jira Alerting improvements (#4829)
add 12dd36c [TE] fix mismatch in dataset names in data availability
listener (#4832)
add a9f6591 Replace NettyTCPServer with QueryServer for server side query
handling (#4834)
add 2189c92 Adding example of pinot-quickstart with Avro encoded messages
in Kafka (#4836)
add 7e292c2 Refactoring realtime segment committer
add 0f331ec Fixing test
add 3a28e74 Addressing comments
add 50902ad Editing test
add 239d7c6 Addressing comments
add 5102de2 Adding committer test
add 229756d Adding test and addressing comments
add 7d9411d Dealing with issues from merge
add 2d5af92 Addressing comments
add cbbf219 Fixing doc
add 8ab0833 Addressing comments
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (89a1cde)
\
N -- N -- N refs/heads/realtimeref (8ab0833)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
README.md | 5 +
docker/images/pinot-presto/Dockerfile | 59 ++
docker/images/pinot-presto/README.md | 61 ++
.../images/{pinot => pinot-presto}/docker-build.sh | 23 +-
.../etc/config.properties} |
[incubator-pinot] branch realtimeref updated (9ea247f -> 89a1cde)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 9ea247f Addressing comments add 89a1cde Addressing comments This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (9ea247f) \ N -- N -- N refs/heads/realtimeref (89a1cde) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../core/data/manager/realtime/SplitSegmentCommitter.java | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (8a63ab3 -> 9ea247f)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 8a63ab3 Fixing doc add 9ea247f Addressing comments No new revisions were added by this update. Summary of changes: .../java/org/apache/pinot/controller/helix/ControllerTest.java | 9 ++--- .../core/data/manager/realtime/DefaultSegmentCommitter.java | 6 +++--- .../core/data/manager/realtime/LLRealtimeSegmentDataManager.java | 5 +++-- .../pinot/core/data/manager/realtime/SegmentCommitter.java | 6 +++--- .../core/data/manager/realtime/SegmentCommitterFactory.java | 4 ++-- .../pinot/core/data/manager/realtime/SplitSegmentCommitter.java | 5 ++--- .../tests/DefaultCommitterRealtimeIntegrationTest.java | 9 + 7 files changed, 20 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (8f19edc -> 8a63ab3)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 8f19edc Addressing comments add 8a63ab3 Fixing doc No new revisions were added by this update. Summary of changes: .../apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java | 1 - 1 file changed, 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (f3489b4 -> 8f19edc)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard f3489b4 Dealing with issues from merge discard c1d9cec Merge branch 'master' into realtimeref discard 0a83fab Adding test and addressing comments discard a91ff14 Adding committer test discard d884868 Addressing comments discard 8f8a06f Editing test discard fc14c1a Addressing comments discard 15e172d Fixing test discard 4e8b535 Refactoring realtime segment committer add 8147714 Adding example for aws (#4789) add aba5482 Data Anonymizer Tool (#4747) add 343aadb Disable compression when numPartitions are less than the threshold (#4797) add 8516fdb [TE] frontend - harleyjj/alert-details - small patch for dimensions column in anomaly table (#4799) add ea3ef34 Add query options from json to existing query options, instead of overriding (#4786) add 6e59583 Refactoring realtime segment committer add be74d25 Fixing test add e655d76 Addressing comments add 30b6e33 Editing test add 6dfc2bb Addressing comments add f662d81 Adding committer test add 85ef958 Adding test and addressing comments add 5d71622 Dealing with issues from merge add 8f19edc Addressing comments This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (f3489b4) \ N -- N -- N refs/heads/realtimeref (8f19edc) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: kubernetes/examples/helm/README.md | 74 +- .../examples/helm/pinot-realtime-quickstart.yml| 36 +- .../requesthandler/BaseBrokerRequestHandler.java | 52 +- .../requesthandler/BrokerRequestOptionsTest.java | 140 +++ .../apache/pinot/common/utils/CommonConstants.java |1 - .../pinot/common/utils/helix/HelixHelper.java | 24 +- .../org/apache/pinot/pql/parsers/Pql2Compiler.java | 23 + .../pql2/ast/ComparisonPredicateAstNode.java |4 + .../pinot/pql/parsers/pql2/ast/SelectAstNode.java | 16 + .../apache/pinot/controller/ControllerStarter.java |4 - .../manager/realtime/DefaultSegmentCommitter.java |6 +- .../realtime/LLRealtimeSegmentDataManager.java | 10 +- .../manager/realtime/SegmentCommitterFactory.java |8 +- .../pinot/core/data/readers/PinotSegmentUtil.java |6 +- .../DefaultCommitterRealtimeIntegrationTest.java | 155 +-- .../tests/LLCRealtimeClusterIntegrationTest.java | 13 +- .../pinot/tools/PinotDataAndQueryAnonymizer.java | 1303 .../pinot/tools/admin/PinotAdministrator.java |4 +- .../tools/admin/command/AnonymizeDataCommand.java | 181 +++ .../pinot/tools/TestDataAndQueryAnonymizer.java| 105 ++ pinot-tools/src/test/resources/C11.dict| 12 + pinot-tools/src/test/resources/columns.mapping | 11 + pinot-tools/src/test/resources/queries.generated |6 + pinot-tools/src/test/resources/queries.raw |6 + .../app/pods/components/alert-details/component.js |8 +- 25 files changed, 2050 insertions(+), 158 deletions(-) create mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/requesthandler/BrokerRequestOptionsTest.java create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/PinotDataAndQueryAnonymizer.java create mode 100644 pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/AnonymizeDataCommand.java create mode 100644 pinot-tools/src/test/java/org/apache/pinot/tools/TestDataAndQueryAnonymizer.java create mode 100644 pinot-tools/src/test/resources/C11.dict create mode 100644 pinot-tools/src/test/resources/columns.mapping create mode 100644 pinot-tools/src/test/resources/queries.generated create mode 100644 pinot-tools/src/test/resources/queries.raw - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (5fdb133 -> f3489b4)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 5fdb133 Dealing with issues from merge add f3489b4 Dealing with issues from merge This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (5fdb133) \ N -- N -- N refs/heads/realtimeref (f3489b4) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../pinot/integration/tests/DefaultCommitterRealtimeIntegrationTest.java | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (c1d9cec -> 5fdb133)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from c1d9cec Merge branch 'master' into realtimeref add 5fdb133 Dealing with issues from merge No new revisions were added by this update. Summary of changes: .../pinot/integration/tests/DefaultCommitterRealtimeIntegrationTest.java | 1 - 1 file changed, 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (0a83fab -> c1d9cec)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 0a83fab Adding test and addressing comments add 6ee6a41 [TE] Add unit test for DetectionEmailAlerter - fix casting bug (#4717) add 3579aae Remove the segment missing error message when acquiring segments (#4714) add a78ac5c [TE][RCA] Fix the broken RCA algorithm tab (#4720) add dc17239 Package parquet and orc record reader into shaded jar (#4716) add f245b5e [TE] support row count metric for Pinot (#4718) add e9e6a33 [TE] frontend - harleyjj/alert-details - handle concurrency errors in alert details component (#4721) add a6f8f1e [Resource Assignment] Plug in resource assignment to LLC REALTIME table (#4695) add 860cb92 Enable full stack trace when there is an exception during tests (#4727) add 6bc6e55 Optimize MutableOffHeapByteArrayStore by directly calling the PinotDataBuffer API (#4362) add c1a0944 [TE] Fix ignored timeout by checking Thread.interrupted (#4690) add 15bde95 [TE] web - harleyjj/rca-session - save dimension-algorithm table settings with session and load them (#4712) add 2f3cd43 [TE] put default settings for subscription YAML (#4723) add f015c68 Add sql query command (#4726) add 1b8cdd8 Fix comparison to pick the right minimum value for hash-set (#4588) add 3a77194 Enhance the exception handling in TableRebalancer (#4734) add ac5ada2 Catch exception when checking lead controller resource is enabled (#4736) add b00b66c [TE] correct time format before send query to Vertica (#4548) add 2f72432 Fix state change when consumption is stopped (#4739) add 9c84de7 Changing error log to warn in SegmentPreprocessingMapper (#4735) add 7d66875 Implement OffHeapMutableBytesStore for real-time var-length bytes store (#4385) add 5af88ef [TE] frontend - harleyjj/alert-details - fix more preview comparison bugs (#4738) add f3b6617 Make Kafka offset out of range as an transient exception. (#4618) add 494ff8a Optimizations for IndexedTable resize (#4728) add a8c6f8d Adding bootstrap mode for Pinot-hadoop job to output segments into relative directories. (#4742) add 3c3c716 Fix a wrong option name for AddSchema in the information log (#4746) add 1f5bf57 Presence vector (#4585) add 74aea85 [SegmentGeneratorConfig Cleanup] Replace checkTimeColumnValidityDuringGeneration with skipTimeValueCheck (#4745) add 04f2b7c AvroSchemaToPinotSchema should return immediately if outputDir doesn't exist (#4751) add 716a3b2 Clean up duplicate/unused metrics in controller (#4750) add 4f849ea [New Transformation function] DateTrunc: Identical semantics to presto's date_trunc (#4740) add 4fc317b First commit, Fixed a typo in ReadMe (#4755) add 5b97d06 fix issue #4115, change replaceAll() to replace (#4116) add b30a692 [TE] frontend - harleyjj/alert-details - comparison revisions and facelift based on feedback (#4749) add 8040b09 [TE][notification] Ability to reopen and reuse existing Jira tickets for alerts (#4748) add 367f7e4 [TE] Merge anomaly properties (#4744) add b48c412 [TE] frontend - harleyjj/alert-details - remove duplicate key from component (#4760) add 07bd379 Bug fixing: Json decoder extracts wrong time field value if schema contains outgoingTimeFieldSpec (#4758) add 5439e77 Helm chart templates for Pinot deployment and quickstart example (#4725) add 6d3c7de Introduce a mode in the server to disable queries (#4767) add 22aa781 [TE] deactivate metrics instead of deleting (#4768) add 2ce7502 Fix the bug of releasing the segment when there are still threads working on it (#4764) add 5d30495 [TE] endpoints - harleyjj/alert-activation - make endpoint for toggling alert activation (#4763) add f0c95dd [TE] Clean up ADContentFormatterContext (#4765) add 4afabca Show all items in the side menu (#4770) add a3977b9 [TE] frontend - harleyjj/self-serve-alert-yaml-details - add toggle button for activating/deactivating alert (#4769) add fa07579 Simplify the constant calculation in CombinePlanNode (#4772) add 4f3d650 [TE] Configure default alerter before constructing properties (#4771) add 19ccfe0 [TE] Extending Jira Merge capabilities for supporting dimensional alerter (#4766) add faa068d [TE] add event trigger listener for scheduling (#4724) add 1c66672 Adding superset demo example (#4779) add 6026746 [TE][notification] Minor tweeking of jira alerter (#4776) add b2b1852 Adding example for azure (#4781) add 0614310 Prepare License and Notice files for 0.2.0 release (#4782) add 23303ec [TE] minor naming change on ThirdEyeAnomalyConfiguration (#4785) add 363d47f [TE] fix the bug
[incubator-pinot] branch realtimeref updated (8f8a06f -> 0a83fab)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 8f8a06f Editing test add d884868 Addressing comments add a91ff14 Adding committer test add 0a83fab Adding test and addressing comments No new revisions were added by this update. Summary of changes: .../apache/pinot/controller/ControllerStarter.java | 4 + .../pinot/controller/helix/ControllerTest.java | 9 +- .../manager/realtime/DefaultSegmentCommitter.java | 7 +- .../realtime/LLRealtimeSegmentDataManager.java | 16 +- .../data/manager/realtime/SegmentCommitter.java| 15 -- .../manager/realtime/SegmentCommitterFactory.java | 30 ++- .../manager/realtime/SplitSegmentCommitter.java| 7 +- pinot-integration-tests/pom.xml| 4 + .../DefaultCommitterRealtimeIntegrationTest.java | 227 + 9 files changed, 260 insertions(+), 59 deletions(-) create mode 100644 pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/DefaultCommitterRealtimeIntegrationTest.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (fc14c1a -> 8f8a06f)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from fc14c1a Addressing comments add 8f8a06f Editing test No new revisions were added by this update. Summary of changes: .../manager/realtime/DefaultSegmentCommitter.java| 2 -- .../realtime/LLRealtimeSegmentDataManager.java | 20 ++-- .../data/manager/realtime/SplitSegmentCommitter.java | 7 +-- .../realtime/LLRealtimeSegmentDataManagerTest.java | 2 +- 4 files changed, 12 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (15e172d -> fc14c1a)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 15e172d Fixing test add fc14c1a Addressing comments No new revisions were added by this update. Summary of changes: .../apache/pinot/common/utils/CommonConstants.java | 1 + .../manager/realtime/DefaultSegmentCommitter.java | 14 -- .../realtime/LLRealtimeSegmentDataManager.java | 22 +++-- .../data/manager/realtime/SegmentCommitter.java| 22 + .../manager/realtime/SegmentCommitterFactory.java | 56 ++ .../manager/realtime/SplitSegmentCommitter.java| 13 +++-- 6 files changed, 116 insertions(+), 12 deletions(-) create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitterFactory.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Changing error log to warn in SegmentPreprocessingMapper (#4735)
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 9c84de7 Changing error log to warn in SegmentPreprocessingMapper
(#4735)
9c84de7 is described below
commit 9c84de7b6fb29c7ede16e7d0462fe1bdfc0fd1fc
Author: Jennifer Dai
AuthorDate: Thu Oct 24 09:48:57 2019 -0700
Changing error log to warn in SegmentPreprocessingMapper (#4735)
---
.../apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java
index 5f2b7ac..cd70235 100644
---
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java
+++
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java
@@ -47,6 +47,7 @@ public class SegmentPreprocessingMapper extends
Mapper, N
private boolean _isAppend = false;
private NormalizedDateSegmentNameGenerator
_normalizedDateSegmentNameGenerator;
private String _sampleNormalizedTimeColumnValue;
+ private boolean _firstInstanceOfMismatchedTime = true;
@Override
public void setup(final Context context) {
@@ -91,10 +92,11 @@ public class SegmentPreprocessingMapper extends
Mapper, N
String timeColumnValue = record.datum().get(_timeColumn).toString();
String normalizedTimeColumnValue =
_normalizedDateSegmentNameGenerator.getNormalizedDate(timeColumnValue);
- if (!normalizedTimeColumnValue.equals(_sampleNormalizedTimeColumnValue))
{
+ if (!normalizedTimeColumnValue.equals(_sampleNormalizedTimeColumnValue)
&& _firstInstanceOfMismatchedTime) {
+_firstInstanceOfMismatchedTime = false;
// TODO: Create a custom exception and gracefully catch this exception
outside, changing what the path to input
// into segment creation should be
-LOGGER.error("This segment contains multiple time units. Sample is {},
current is {}", _sampleNormalizedTimeColumnValue, normalizedTimeColumnValue);
+LOGGER.warn("This segment contains multiple time units. Sample is {},
current is {}", _sampleNormalizedTimeColumnValue, normalizedTimeColumnValue);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch log updated (d4c3362 -> bdf968c)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch log in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from d4c3362 Changing error log to warn add bdf968c Making sure log warns only once No new revisions were added by this update. Summary of changes: .../apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Changing error log to warn
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch log
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit d4c33626c37de81dfba99bf5bdb03390bff753dc
Author: Jennifer Dai
AuthorDate: Tue Oct 22 10:32:53 2019 -0700
Changing error log to warn
---
.../org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java
index 5f2b7ac..0b51c6f 100644
---
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java
+++
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java
@@ -94,7 +94,7 @@ public class SegmentPreprocessingMapper extends
Mapper, N
if (!normalizedTimeColumnValue.equals(_sampleNormalizedTimeColumnValue))
{
// TODO: Create a custom exception and gracefully catch this exception
outside, changing what the path to input
// into segment creation should be
-LOGGER.error("This segment contains multiple time units. Sample is {},
current is {}", _sampleNormalizedTimeColumnValue, normalizedTimeColumnValue);
+LOGGER.warn("This segment contains multiple time units. Sample is {},
current is {}", _sampleNormalizedTimeColumnValue, normalizedTimeColumnValue);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch log created (now d4c3362)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch log in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at d4c3362 Changing error log to warn This branch includes the following new commits: new d4c3362 Changing error log to warn The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch surefire updated (f3b0ba5 -> 6f99af2)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch surefire in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard f3b0ba5 Updating maven-surefire-plugin version add 6f99af2 Updating maven-surefire-plugin version This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (f3b0ba5) \ N -- N -- N refs/heads/surefire (6f99af2) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch surefire created (now f3b0ba5)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch surefire in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at f3b0ba5 Updating maven-surefire-plugin version No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch hybrid created (now af6e461)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch hybrid in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at af6e461 Adding logs for HybridClusterIntegrationTest This branch includes the following new commits: new af6e461 Adding logs for HybridClusterIntegrationTest The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Adding logs for HybridClusterIntegrationTest
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch hybrid
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit af6e461a414bc28518e11097d1406496036054b7
Author: Jennifer Dai
AuthorDate: Mon Oct 21 15:54:27 2019 -0700
Adding logs for HybridClusterIntegrationTest
---
.../test/java/org/apache/pinot/controller/helix/ControllerTest.java | 3 +++
.../pinot/integration/tests/BaseClusterIntegrationTestSet.java | 5 +
.../apache/pinot/integration/tests/HybridClusterIntegrationTest.java | 2 ++
3 files changed, 10 insertions(+)
diff --git
a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTest.java
b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTest.java
index 94edddb..ddca7d7 100644
---
a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTest.java
+++
b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/ControllerTest.java
@@ -111,6 +111,8 @@ public abstract class ControllerTest {
private ZkStarter.ZookeeperInstance _zookeeperInstance;
+ private Logger LOGGER = LoggerFactory.getLogger(ControllerTest.class);
+
protected String getHelixClusterName() {
return getClass().getSimpleName();
}
@@ -127,6 +129,7 @@ public abstract class ControllerTest {
try {
ZkStarter.stopLocalZkServer(_zookeeperInstance);
} catch (Exception e) {
+ LOGGER.info("Could not stop local zk server");
// Swallow exceptions
}
}
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
index 83b46ba..f78c6827 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/BaseClusterIntegrationTestSet.java
@@ -41,6 +41,8 @@ import org.apache.pinot.common.config.TableNameBuilder;
import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.JsonUtils;
import org.apache.pinot.util.TestUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.testng.Assert;
@@ -57,6 +59,8 @@ public abstract class BaseClusterIntegrationTestSet extends
BaseClusterIntegrati
private static final int DEFAULT_NUM_QUERIES_TO_GENERATE = 100;
private static final int DEFAULT_MAX_NUM_QUERIES_TO_SKIP_IN_QUERY_FILE = 200;
+ protected static final Logger LOGGER =
LoggerFactory.getLogger(BaseClusterIntegrationTest.class.getName());
+
/**
* Can be overridden to change default setting
*/
@@ -225,6 +229,7 @@ public abstract class BaseClusterIntegrationTestSet extends
BaseClusterIntegrati
getPinotConnection().execute("select $docId, $segmentName, $hostName from
mytable where $docId = 5 limit 50");
getPinotConnection().execute("select $docId, $segmentName, $hostName from
mytable where $docId > 19998 limit 50");
getPinotConnection().execute("select max($docId) from mytable group by
$segmentName");
+LOGGER.info("Successfully completed virtual column query test");
}
/**
diff --git
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTest.java
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTest.java
index ae6753d..d574467 100644
---
a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTest.java
+++
b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/HybridClusterIntegrationTest.java
@@ -282,7 +282,9 @@ public class HybridClusterIntegrationTest extends
BaseClusterIntegrationTestSet
// Try deleting the tables and check that they have no routing table
final String tableName = getTableName();
dropOfflineTable(tableName);
+LOGGER.info("Dropped offline table successfully");
dropRealtimeTable(tableName);
+LOGGER.info("Dropped realtime table successfully");
// Routing table should not have any entries (length = 0) after deleting
all tables
TestUtils.waitForCondition(new Function() {
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref updated (4e8b535 -> 15e172d)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 4e8b535 Refactoring realtime segment committer add 15e172d Fixing test No new revisions were added by this update. Summary of changes: .../pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java | 2 +- .../core/data/manager/realtime/LLRealtimeSegmentDataManagerTest.java| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimeref created (now 4e8b535)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimeref in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 4e8b535 Refactoring realtime segment committer This branch includes the following new commits: new 4e8b535 Refactoring realtime segment committer The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Refactoring realtime segment committer
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch realtimeref
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 4e8b535c84e4d029d85fa50db0e704b30c700df5
Author: Jennifer Dai
AuthorDate: Wed Oct 16 11:05:21 2019 -0700
Refactoring realtime segment committer
---
.../manager/realtime/DefaultSegmentCommitter.java | 54 ++
.../realtime/LLRealtimeSegmentDataManager.java | 84 -
.../data/manager/realtime/SegmentCommitter.java| 29
.../manager/realtime/SplitSegmentCommitter.java| 86 ++
4 files changed, 183 insertions(+), 70 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/DefaultSegmentCommitter.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/DefaultSegmentCommitter.java
new file mode 100644
index 000..84bde7a
--- /dev/null
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/DefaultSegmentCommitter.java
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import java.io.File;
+import org.apache.pinot.common.protocols.SegmentCompletionProtocol;
+import org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler;
+import org.slf4j.Logger;
+
+
+/**
+ * Sends segmentCommit() to the controller.
+ * If that succeeds, swap in-memory segment with the one built.
+ */
+public class DefaultSegmentCommitter implements SegmentCommitter{
+ private SegmentCompletionProtocol.Request.Params _params;
+ private ServerSegmentCompletionProtocolHandler _protocolHandler;
+
+ private Logger _segmentLogger;
+
+ public DefaultSegmentCommitter(SegmentCompletionProtocol.Request.Params
params, ServerSegmentCompletionProtocolHandler protocolHandler, Logger
segmentLogger) {
+_params = params;
+_protocolHandler = protocolHandler;
+
+_segmentLogger = segmentLogger;
+ }
+
+ @Override
+ public SegmentCompletionProtocol.Response commit(long currentOffset, int
numRows, LLRealtimeSegmentDataManager.SegmentBuildDescriptor
segmentBuildDescriptor) {
+final File segmentTarFile = new
File(segmentBuildDescriptor.getSegmentTarFilePath());
+
+SegmentCompletionProtocol.Response response =
_protocolHandler.segmentCommit(_params, segmentTarFile);
+if
(!response.getStatus().equals(SegmentCompletionProtocol.ControllerResponseStatus.COMMIT_SUCCESS))
{
+ _segmentLogger.warn("Commit failed with response {}",
response.toJsonString());
+}
+return response;
+ }
+}
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
index 7bfb348..5c286bb 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
@@ -759,58 +759,6 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
}
}
- private SegmentCompletionProtocol.Response
doSplitCommit(SegmentCompletionProtocol.Response prevResponse) {
-final File segmentTarFile = new
File(_segmentBuildDescriptor.getSegmentTarFilePath());
-SegmentCompletionProtocol.Request.Params params = new
SegmentCompletionProtocol.Request.Params();
-
-
params.withSegmentName(_segmentNameStr).withOffset(_currentOffset).withNumRows(_numRowsConsumed)
-
.withInstanceId(_instanceId).withBuildTimeMillis(_segmentBuildDescriptor.getBuildTimeMillis())
-.withSegmentSizeBytes(_segmentBuildDescriptor.getSegmentSizeBytes())
-.withWaitTimeMillis(_segmentBuildDescriptor.getWaitTimeMillis());
-if (_isOffHeap) {
- params.withMemoryUsedBytes(_memoryManager.getTotalAllocatedBytes());
-}
-SegmentCompletionProtocol.Response segmentCommitStartResponse =
_protocolHandler.segmentCommitStart(params);
-if (!segmentCommitStartResponse
[incubator-pinot] branch deletepreprocessoutput deleted (was 24d551c)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch deletepreprocessoutput in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 24d551c Deleting preprocess output post segment creation The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch paths deleted (was fa31be6)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch paths in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was fa31be6 Disallowing multiple inputs in preprocess The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch preprocess deleted (was 6b9e3aa)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 6b9e3aa Addressing comments The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (ad391dd -> bb5913f)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from ad391dd Enable distinct and throw exceptions for not-supported cases (#4648) add bb5913f Enabling alternative controller rest API classes in preprocess (#4646) No new revisions were added by this update. Summary of changes: .../apache/pinot/hadoop/job/BaseSegmentJob.java| 46 +++ .../pinot/hadoop/job/SegmentCreationJob.java | 18 +--- .../pinot/hadoop/job/SegmentPreprocessingJob.java | 51 -- 3 files changed, 74 insertions(+), 41 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch preprocess updated (f190f36 -> 6b9e3aa)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from f190f36 Reverting to previous state and adding proper schema functionality add 6b9e3aa Addressing comments No new revisions were added by this update. Summary of changes: .../java/org/apache/pinot/hadoop/job/BaseSegmentJob.java | 1 - .../apache/pinot/hadoop/job/SegmentPreprocessingJob.java | 15 --- 2 files changed, 16 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch preprocess updated (67bcabd -> f190f36)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 67bcabd Addressing comments add dd94747 Revert "Addressing comments" add f190f36 Reverting to previous state and adding proper schema functionality No new revisions were added by this update. Summary of changes: .../apache/pinot/hadoop/job/BaseSegmentJob.java| 19 --- .../pinot/hadoop/job/SegmentCreationJob.java | 16 + .../pinot/hadoop/job/SegmentPreprocessingJob.java | 27 ++ 3 files changed, 42 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch preprocess updated (3e723f5 -> 67bcabd)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 3e723f5 Addressing comments add 67bcabd Addressing comments No new revisions were added by this update. Summary of changes: .../org/apache/pinot/hadoop/job/BaseSegmentJob.java | 19 ++- .../apache/pinot/hadoop/job/SegmentCreationJob.java | 16 .../pinot/hadoop/job/SegmentPreprocessingJob.java | 15 --- 3 files changed, 10 insertions(+), 40 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adldoc deleted (was fbc64b0)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch adldoc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was fbc64b0 Adding pointer to example upload command The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (bfb518e -> b24816a)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from bfb518e Add integration test for ControllerLeaderLocator (#4567) add b24816a Adding sample ADL settings to documentation (#4645) No new revisions were added by this update. Summary of changes: docs/pluggable_storage.rst | 42 -- docs/segment_fetcher.rst | 2 ++ 2 files changed, 42 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch preprocess updated (47ba511 -> 3e723f5)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 47ba511 Removing breaking preconditions check add 3e723f5 Addressing comments No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java | 1 - .../java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch preprocess updated (ced9c58 -> 47ba511)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from ced9c58 Updating schema method with comments add 47ba511 Removing breaking preconditions check No new revisions were added by this update. Summary of changes: .../main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java | 1 - 1 file changed, 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch preprocess updated (7a1a5aa -> ced9c58)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 7a1a5aa Enabling alternative controller rest API classes in preprocess add ced9c58 Updating schema method with comments No new revisions were added by this update. Summary of changes: .../java/org/apache/pinot/hadoop/job/BaseSegmentJob.java| 13 + .../org/apache/pinot/hadoop/job/SegmentCreationJob.java | 1 + .../apache/pinot/hadoop/job/SegmentPreprocessingJob.java| 1 + 3 files changed, 15 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adldoc updated (ed14229 -> fbc64b0)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch adldoc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from ed14229 Adding exact param names add fbc64b0 Adding pointer to example upload command No new revisions were added by this update. Summary of changes: docs/pluggable_storage.rst | 47 -- docs/segment_fetcher.rst | 2 ++ 2 files changed, 35 insertions(+), 14 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch preprocess updated (ca51acc -> 7a1a5aa)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard ca51acc Enabling alternative controller rest API classes in preprocess discard fc2c98b Preprocess changes new 7a1a5aa Enabling alternative controller rest API classes in preprocess This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (ca51acc) \ N -- N -- N refs/heads/preprocess (7a1a5aa) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../.On_Time_On_Time_Performance_2014_10.avro.crc | Bin 41552 -> 0 bytes .../On_Time_On_Time_Performance_2014_10.avro | Bin 5317292 -> 0 bytes .../AirTime.dict | Bin 1444 -> 0 bytes .../AirTime.sv.unsorted.fwd | Bin 10454 -> 0 bytes .../ArrDel15.dict | Bin 12 -> 0 bytes .../ArrDel15.sv.unsorted.fwd | Bin 2323 -> 0 bytes .../DaysSinceEpoch.dict | Bin 124 -> 0 bytes .../DaysSinceEpoch.sv.sorted.fwd | Bin 248 -> 0 bytes .../DepTime.dict | Bin 4516 -> 0 bytes .../DepTime.sv.unsorted.fwd | Bin 12777 -> 0 bytes .../DepTimeBlk.dict | 1 - .../DepTimeBlk.sv.unsorted.fwd| Bin 5808 -> 0 bytes .../DestCityMarketID.dict | Bin 1048 -> 0 bytes .../DestCityMarketID.sv.unsorted.fwd | Bin 10454 -> 0 bytes .../DivActualElapsedTime.dict | Bin 1388 -> 0 bytes .../DivActualElapsedTime.sv.unsorted.fwd | Bin 10454 -> 0 bytes .../DivAirportSeqIDs.dict | Bin 600 -> 0 bytes .../DivAirportSeqIDs.mv.fwd | Bin 52360 -> 0 bytes .../DivTotalGTimes.dict | Bin 1040 -> 0 bytes .../DivTotalGTimes.mv.fwd | Bin 52360 -> 0 bytes .../DivWheelsOffs.dict| Bin 2068 -> 0 bytes .../DivWheelsOffs.mv.fwd | Bin 63975 -> 0 bytes .../DivWheelsOns.dict | Bin 2348 -> 0 bytes .../DivWheelsOns.mv.fwd | Bin 63975 -> 0 bytes .../FlightNum.dict| Bin 18188 -> 0 bytes .../FlightNum.sv.unsorted.fwd | Bin 15100 -> 0 bytes .../LateAircraftDelay.dict| Bin 616 -> 0 bytes .../LateAircraftDelay.sv.unsorted.fwd | Bin 9292 -> 0 bytes .../tmp-1568139829380--6120390252134451091/Month.dict | Bin 4 -> 0 bytes .../Origin.dict | 1 - .../Origin.sv.unsorted.fwd| Bin 10454 -> 0 bytes .../Quarter.dict | Bin 4 -> 0 bytes .../Quarter.sv.sorted.fwd | Bin 8 -> 0 bytes 33 files changed, 2 deletions(-) delete mode 100644 pinot-integration-tests/pinot_hadoop_tmp/inputData/.On_Time_On_Time_Performance_2014_10.avro.crc delete mode 100644 pinot-integration-tests/pinot_hadoop_tmp/inputData/On_Time_On_Time_Performance_2014_10.avro delete mode 100644 pinot-integration-tests/pinot_hadoop_tmp/segments/tmp-1568139829380--6120390252134451091/AirTime.dict delete mode 100644 pinot-integration-tests/pinot_hadoop_tmp/segments/tmp-1568139829380--6120390252134451091/AirTime.sv.unsorted.fwd delete mode 100644 pinot-integration-tests/pinot_hadoop_tmp/segments/tmp-1568139829380--6120390252134451091/ArrDel15.dict delete mode 100644 pinot-integration-tests/pinot_hadoop_tmp/segments/tmp-1568139829380--6120390252134451091/ArrDel15.sv.unsorted.fwd delete mode 100644 pinot-integration-tests/pinot_hadoop_tmp/segments/tmp-1568139829380--6120390252134451091/DaysSinceEpoch.dict delete mode 100644 pinot-integration-tests/pinot_hadoop_tmp/segments/tmp-1568139829380--6120
[incubator-pinot] 01/01: Enabling alternative controller rest API classes in preprocess
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch preprocess
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 7a1a5aaaba3b238d343123e7694cf0f7667b2165
Author: Jennifer Dai
AuthorDate: Wed Sep 25 10:14:40 2019 -0700
Enabling alternative controller rest API classes in preprocess
---
.../apache/pinot/hadoop/job/BaseSegmentJob.java| 35 +++
.../pinot/hadoop/job/SegmentCreationJob.java | 17 -
.../pinot/hadoop/job/SegmentPreprocessingJob.java | 40 ++
3 files changed, 68 insertions(+), 24 deletions(-)
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java
index 488c587..2d72fbb 100644
--- a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java
+++ b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java
@@ -18,17 +18,21 @@
*/
package org.apache.pinot.hadoop.job;
+import com.google.common.base.Preconditions;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import javax.annotation.Nullable;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.pinot.common.Utils;
+import org.apache.pinot.common.config.TableConfig;
+import org.apache.pinot.hadoop.utils.PushLocation;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -37,6 +41,8 @@ public abstract class BaseSegmentJob extends Configured {
protected final Logger _logger = LoggerFactory.getLogger(getClass());
protected final Properties _properties;
protected final Configuration _conf;
+ protected final List _pushLocations;
+ protected final String _rawTableName;
protected BaseSegmentJob(Properties properties) {
_properties = properties;
@@ -44,6 +50,35 @@ public abstract class BaseSegmentJob extends Configured {
setConf(_conf);
Utils.logVersions();
logProperties();
+
+// Optional push location and table parameters. If set, will use the table
config and schema from the push hosts.
+String pushHostsString =
_properties.getProperty(JobConfigConstants.PUSH_TO_HOSTS);
+String pushPortString =
_properties.getProperty(JobConfigConstants.PUSH_TO_PORT);
+if (pushHostsString != null && pushPortString != null) {
+ _pushLocations =
+ PushLocation.getPushLocations(StringUtils.split(pushHostsString,
','), Integer.parseInt(pushPortString));
+} else {
+ _pushLocations = null;
+}
+
+_rawTableName =
Preconditions.checkNotNull(_properties.getProperty(JobConfigConstants.SEGMENT_TABLE_NAME));
+
+ }
+
+ @Nullable
+ protected TableConfig getTableConfig()
+ throws IOException {
+try (ControllerRestApi controllerRestApi = getControllerRestApi()) {
+ return controllerRestApi != null ? controllerRestApi.getTableConfig() :
null;
+}
+ }
+
+ /**
+ * Can be overridden to provide custom controller Rest API.
+ */
+ @Nullable
+ protected ControllerRestApi getControllerRestApi() {
+return _pushLocations != null ? new
DefaultControllerRestApi(_pushLocations, _rawTableName) : null;
}
protected void logProperties() {
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
index aa68f10..427095f 100644
---
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
+++
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
@@ -26,7 +26,6 @@ import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.UUID;
-import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
@@ -194,14 +193,6 @@ public class SegmentCreationJob extends BaseSegmentJob {
_fileSystem.delete(_stagingDir, true);
}
- @Nullable
- protected TableConfig getTableConfig()
- throws IOException {
-try (ControllerRestApi controllerRestApi = getControllerRestApi()) {
- return controllerRestApi != null ? controllerRestApi.getTableConfig() :
null;
-}
- }
-
protected Schema getSchema()
throws IOException {
try (ControllerRestApi controllerRestApi = getControllerRestApi()) {
@@ -215,14 +206,6 @@ public class SegmentCreationJob extends BaseSegmentJob {
}
}
- /**
- * Can be overridden to provide custom controller Rest API.
- */
- @Nullable
- protected ControllerRestApi getControllerRestApi() {
-return _pushLocations != null ? new
DefaultControlle
[incubator-pinot] 02/02: Enabling alternative controller rest API classes in preprocess
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch preprocess
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit ca51acce57eab0a40ba6fbf990499a37abfac6fc
Author: Jennifer Dai
AuthorDate: Wed Sep 25 10:12:07 2019 -0700
Enabling alternative controller rest API classes in preprocess
---
.../apache/pinot/hadoop/job/BaseSegmentJob.java| 35 ++
.../pinot/hadoop/job/SegmentCreationJob.java | 17 ---
.../pinot/hadoop/job/SegmentPreprocessingJob.java | 25 +---
3 files changed, 42 insertions(+), 35 deletions(-)
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java
index 488c587..2d72fbb 100644
--- a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java
+++ b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/BaseSegmentJob.java
@@ -18,17 +18,21 @@
*/
package org.apache.pinot.hadoop.job;
+import com.google.common.base.Preconditions;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import javax.annotation.Nullable;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.pinot.common.Utils;
+import org.apache.pinot.common.config.TableConfig;
+import org.apache.pinot.hadoop.utils.PushLocation;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -37,6 +41,8 @@ public abstract class BaseSegmentJob extends Configured {
protected final Logger _logger = LoggerFactory.getLogger(getClass());
protected final Properties _properties;
protected final Configuration _conf;
+ protected final List _pushLocations;
+ protected final String _rawTableName;
protected BaseSegmentJob(Properties properties) {
_properties = properties;
@@ -44,6 +50,35 @@ public abstract class BaseSegmentJob extends Configured {
setConf(_conf);
Utils.logVersions();
logProperties();
+
+// Optional push location and table parameters. If set, will use the table
config and schema from the push hosts.
+String pushHostsString =
_properties.getProperty(JobConfigConstants.PUSH_TO_HOSTS);
+String pushPortString =
_properties.getProperty(JobConfigConstants.PUSH_TO_PORT);
+if (pushHostsString != null && pushPortString != null) {
+ _pushLocations =
+ PushLocation.getPushLocations(StringUtils.split(pushHostsString,
','), Integer.parseInt(pushPortString));
+} else {
+ _pushLocations = null;
+}
+
+_rawTableName =
Preconditions.checkNotNull(_properties.getProperty(JobConfigConstants.SEGMENT_TABLE_NAME));
+
+ }
+
+ @Nullable
+ protected TableConfig getTableConfig()
+ throws IOException {
+try (ControllerRestApi controllerRestApi = getControllerRestApi()) {
+ return controllerRestApi != null ? controllerRestApi.getTableConfig() :
null;
+}
+ }
+
+ /**
+ * Can be overridden to provide custom controller Rest API.
+ */
+ @Nullable
+ protected ControllerRestApi getControllerRestApi() {
+return _pushLocations != null ? new
DefaultControllerRestApi(_pushLocations, _rawTableName) : null;
}
protected void logProperties() {
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
index aa68f10..427095f 100644
---
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
+++
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
@@ -26,7 +26,6 @@ import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.UUID;
-import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
@@ -194,14 +193,6 @@ public class SegmentCreationJob extends BaseSegmentJob {
_fileSystem.delete(_stagingDir, true);
}
- @Nullable
- protected TableConfig getTableConfig()
- throws IOException {
-try (ControllerRestApi controllerRestApi = getControllerRestApi()) {
- return controllerRestApi != null ? controllerRestApi.getTableConfig() :
null;
-}
- }
-
protected Schema getSchema()
throws IOException {
try (ControllerRestApi controllerRestApi = getControllerRestApi()) {
@@ -215,14 +206,6 @@ public class SegmentCreationJob extends BaseSegmentJob {
}
}
- /**
- * Can be overridden to provide custom controller Rest API.
- */
- @Nullable
- protected ControllerRestApi getControllerRestApi() {
-return _pushLocations != null ? new
DefaultControlle
[incubator-pinot] 01/02: Preprocess changes
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch preprocess
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit fc2c98b517cf4ae297176a5e6c54c3cd820664f2
Author: Jennifer Dai
AuthorDate: Mon Sep 23 13:33:54 2019 -0700
Preprocess changes
---
.../pinot/hadoop/job/SegmentPreprocessingJob.java | 51 ++---
.../.On_Time_On_Time_Performance_2014_10.avro.crc | Bin 0 -> 41552 bytes
.../On_Time_On_Time_Performance_2014_10.avro | Bin 0 -> 5317292 bytes
.../AirTime.dict | Bin 0 -> 1444 bytes
.../AirTime.sv.unsorted.fwd| Bin 0 -> 10454 bytes
.../ArrDel15.dict | Bin 0 -> 12 bytes
.../ArrDel15.sv.unsorted.fwd | Bin 0 -> 2323 bytes
.../DaysSinceEpoch.dict| Bin 0 -> 124 bytes
.../DaysSinceEpoch.sv.sorted.fwd | Bin 0 -> 248 bytes
.../DepTime.dict | Bin 0 -> 4516 bytes
.../DepTime.sv.unsorted.fwd| Bin 0 -> 12777 bytes
.../DepTimeBlk.dict| 1 +
.../DepTimeBlk.sv.unsorted.fwd | Bin 0 -> 5808 bytes
.../DestCityMarketID.dict | Bin 0 -> 1048 bytes
.../DestCityMarketID.sv.unsorted.fwd | Bin 0 -> 10454 bytes
.../DivActualElapsedTime.dict | Bin 0 -> 1388 bytes
.../DivActualElapsedTime.sv.unsorted.fwd | Bin 0 -> 10454 bytes
.../DivAirportSeqIDs.dict | Bin 0 -> 600 bytes
.../DivAirportSeqIDs.mv.fwd| Bin 0 -> 52360 bytes
.../DivTotalGTimes.dict| Bin 0 -> 1040 bytes
.../DivTotalGTimes.mv.fwd | Bin 0 -> 52360 bytes
.../DivWheelsOffs.dict | Bin 0 -> 2068 bytes
.../DivWheelsOffs.mv.fwd | Bin 0 -> 63975 bytes
.../DivWheelsOns.dict | Bin 0 -> 2348 bytes
.../DivWheelsOns.mv.fwd| Bin 0 -> 63975 bytes
.../FlightNum.dict | Bin 0 -> 18188 bytes
.../FlightNum.sv.unsorted.fwd | Bin 0 -> 15100 bytes
.../LateAircraftDelay.dict | Bin 0 -> 616 bytes
.../LateAircraftDelay.sv.unsorted.fwd | Bin 0 -> 9292 bytes
.../Month.dict | Bin 0 -> 4 bytes
.../Origin.dict| 1 +
.../Origin.sv.unsorted.fwd | Bin 0 -> 10454 bytes
.../Quarter.dict | Bin 0 -> 4 bytes
.../Quarter.sv.sorted.fwd | Bin 0 -> 8 bytes
34 files changed, 46 insertions(+), 7 deletions(-)
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
index 942fb6d..89946ed 100644
---
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
+++
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
@@ -27,6 +27,7 @@ import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.zip.GZIPInputStream;
+import javax.annotation.Nullable;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileStream;
import org.apache.avro.generic.GenericDatumReader;
@@ -87,7 +88,7 @@ public class SegmentPreprocessingJob extends BaseSegmentJob {
private final Path _inputSegmentDir;
private final Path _preprocessedOutputDir;
protected final String _rawTableName;
- protected final List _pushLocations;
+ protected List _pushLocations;
// Optional.
private final Path _pathToDependencyJar;
@@ -115,9 +116,7 @@ public class SegmentPreprocessingJob extends BaseSegmentJob
{
_pushLocations =
PushLocation.getPushLocations(StringUtils.split(pushHostsString,
','), Integer.parseInt(pushPortString));
} else {
- throw new RuntimeException(String
- .format("Push location is mis-configured! %s: %s, %s: %s",
JobConfigConstants.PUSH_TO_HOSTS, pushHostsString,
- JobConfigConstants.PUSH_TO_PORT, pushPortString));
+ _pushLocations = null;
}
_logger.info("*");
@@ -375,13 +374,51 @@ public class SegmentPreprocessingJob extends
BaseSegmentJob {
fieldSet.add(hashCodeField);
}
+ /**
+ * Can be overridden to provide custom controller Rest API.
+ */
+ @Nullable
+ protected ControllerRestApi getControllerRestApi() {
+return _pushLocations != null ? new
DefaultControllerRestApi(_pushLocations, _rawTabl
[incubator-pinot] branch preprocess updated (6cb86b3 -> ca51acc)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 6cb86b3 Preprocess changes new fc2c98b Preprocess changes new ca51acc Enabling alternative controller rest API classes in preprocess This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (6cb86b3) \ N -- N -- N refs/heads/preprocess (ca51acc) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/pinot/hadoop/job/BaseSegmentJob.java| 35 ++ .../pinot/hadoop/job/SegmentCreationJob.java | 17 --- .../pinot/hadoop/job/SegmentPreprocessingJob.java | 25 3 files changed, 55 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adldoc updated (df73e8c -> ed14229)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch adldoc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from df73e8c Adding adl param examples add ed14229 Adding exact param names No new revisions were added by this update. Summary of changes: docs/pluggable_storage.rst | 16 1 file changed, 16 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Adding adl param examples
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a commit to branch adldoc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit df73e8c815e4dd39da9dee025cf73bd7def99b54 Author: Jennifer Dai AuthorDate: Wed Sep 25 09:36:34 2019 -0700 Adding adl param examples --- docs/pluggable_storage.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/pluggable_storage.rst b/docs/pluggable_storage.rst index c0aef40..90618fc 100644 --- a/docs/pluggable_storage.rst +++ b/docs/pluggable_storage.rst @@ -57,6 +57,9 @@ AzurePinotFS requires the following configs according to your environment: adl.accountId, adl.authEndpoint, adl.clientId, adl.clientSecret +You can find the parameters in your account as follows: +https://stackoverflow.com/questions/56349040/what-is-clientid-authtokenendpoint-clientkey-for-accessing-azure-data-lake + Please also make sure to set the following config with the value "adl" segment.fetcher.protocols : adl - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adldoc updated (aa1e0b8 -> df73e8c)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch adldoc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard aa1e0b8 Adding adl param examples new df73e8c Adding adl param examples This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (aa1e0b8) \ N -- N -- N refs/heads/adldoc (df73e8c) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/pluggable_storage.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Adding adl param examples
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a commit to branch adldoc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit aa1e0b822a73ae83772699b2355ed65eec8545fa Author: Jennifer Dai AuthorDate: Wed Sep 25 09:36:34 2019 -0700 Adding adl param examples --- docs/pluggable_storage.rst | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/pluggable_storage.rst b/docs/pluggable_storage.rst index c0aef40..875faf5 100644 --- a/docs/pluggable_storage.rst +++ b/docs/pluggable_storage.rst @@ -1,4 +1,4 @@ -.. +\.. .. Licensed to the Apache Software Foundation (ASF) under one .. or more contributor license agreements. See the NOTICE file .. distributed with this work for additional information @@ -57,6 +57,9 @@ AzurePinotFS requires the following configs according to your environment: adl.accountId, adl.authEndpoint, adl.clientId, adl.clientSecret +You can find the parameters in your account as follows: +https://stackoverflow.com/questions/56349040/what-is-clientid-authtokenendpoint-clientkey-for-accessing-azure-data-lake + Please also make sure to set the following config with the value "adl" segment.fetcher.protocols : adl - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adldoc created (now aa1e0b8)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch adldoc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at aa1e0b8 Adding adl param examples This branch includes the following new commits: new aa1e0b8 Adding adl param examples The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch preprocess created (now 6cb86b3)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch preprocess in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 6cb86b3 Preprocess changes This branch includes the following new commits: new 6cb86b3 Preprocess changes The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Preprocess changes
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch preprocess
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 6cb86b3303ec686911878bfd7ce538a0fdb686e1
Author: Jennifer Dai
AuthorDate: Mon Sep 23 13:33:54 2019 -0700
Preprocess changes
---
.../pinot/hadoop/job/SegmentPreprocessingJob.java | 25 +++--
.../.On_Time_On_Time_Performance_2014_10.avro.crc | Bin 0 -> 41552 bytes
.../On_Time_On_Time_Performance_2014_10.avro | Bin 0 -> 5317292 bytes
.../AirTime.dict | Bin 0 -> 1444 bytes
.../AirTime.sv.unsorted.fwd| Bin 0 -> 10454 bytes
.../ArrDel15.dict | Bin 0 -> 12 bytes
.../ArrDel15.sv.unsorted.fwd | Bin 0 -> 2323 bytes
.../DaysSinceEpoch.dict| Bin 0 -> 124 bytes
.../DaysSinceEpoch.sv.sorted.fwd | Bin 0 -> 248 bytes
.../DepTime.dict | Bin 0 -> 4516 bytes
.../DepTime.sv.unsorted.fwd| Bin 0 -> 12777 bytes
.../DepTimeBlk.dict| 1 +
.../DepTimeBlk.sv.unsorted.fwd | Bin 0 -> 5808 bytes
.../DestCityMarketID.dict | Bin 0 -> 1048 bytes
.../DestCityMarketID.sv.unsorted.fwd | Bin 0 -> 10454 bytes
.../DivActualElapsedTime.dict | Bin 0 -> 1388 bytes
.../DivActualElapsedTime.sv.unsorted.fwd | Bin 0 -> 10454 bytes
.../DivAirportSeqIDs.dict | Bin 0 -> 600 bytes
.../DivAirportSeqIDs.mv.fwd| Bin 0 -> 52360 bytes
.../DivTotalGTimes.dict| Bin 0 -> 1040 bytes
.../DivTotalGTimes.mv.fwd | Bin 0 -> 52360 bytes
.../DivWheelsOffs.dict | Bin 0 -> 2068 bytes
.../DivWheelsOffs.mv.fwd | Bin 0 -> 63975 bytes
.../DivWheelsOns.dict | Bin 0 -> 2348 bytes
.../DivWheelsOns.mv.fwd| Bin 0 -> 63975 bytes
.../FlightNum.dict | Bin 0 -> 18188 bytes
.../FlightNum.sv.unsorted.fwd | Bin 0 -> 15100 bytes
.../LateAircraftDelay.dict | Bin 0 -> 616 bytes
.../LateAircraftDelay.sv.unsorted.fwd | Bin 0 -> 9292 bytes
.../Month.dict | Bin 0 -> 4 bytes
.../Origin.dict| 1 +
.../Origin.sv.unsorted.fwd | Bin 0 -> 10454 bytes
.../Quarter.dict | Bin 0 -> 4 bytes
.../Quarter.sv.sorted.fwd | Bin 0 -> 8 bytes
34 files changed, 20 insertions(+), 7 deletions(-)
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
index 942fb6d..83381fc 100644
---
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
+++
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
@@ -27,6 +27,7 @@ import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.zip.GZIPInputStream;
+import javax.annotation.Nullable;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileStream;
import org.apache.avro.generic.GenericDatumReader;
@@ -87,7 +88,7 @@ public class SegmentPreprocessingJob extends BaseSegmentJob {
private final Path _inputSegmentDir;
private final Path _preprocessedOutputDir;
protected final String _rawTableName;
- protected final List _pushLocations;
+ protected List _pushLocations;
// Optional.
private final Path _pathToDependencyJar;
@@ -115,9 +116,7 @@ public class SegmentPreprocessingJob extends BaseSegmentJob
{
_pushLocations =
PushLocation.getPushLocations(StringUtils.split(pushHostsString,
','), Integer.parseInt(pushPortString));
} else {
- throw new RuntimeException(String
- .format("Push location is mis-configured! %s: %s, %s: %s",
JobConfigConstants.PUSH_TO_HOSTS, pushHostsString,
- JobConfigConstants.PUSH_TO_PORT, pushPortString));
+ _pushLocations = null;
}
_logger.info("*");
@@ -375,13 +374,25 @@ public class SegmentPreprocessingJob extends
BaseSegmentJob {
fieldSet.add(hashCodeField);
}
+ /**
+ * Can be overridden to provide custom controller Rest API.
+ */
+ @Nullable
+ protected ControllerRestApi getControllerRestApi() {
+return _pushLocations != null ? new
DefaultControllerRestApi(_pushLocations, _rawTableNam
[incubator-pinot] branch revert deleted (was 353c7b1)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch revert in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 353c7b1 Revert "Set processingException when all queried segments cannot be acquired (#3942)" The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (31594da -> 283a495)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 31594da Replace Hashmap to Object2IntOpenHashMap in OnHeapStringDictionary (#4568) add 283a495 Revert "Set processingException when all queried segments cannot be acquired (#3942)" (#4632) No new revisions were added by this update. Summary of changes: .../pinot/common/exception/QueryException.java | 2 - .../core/data/manager/BaseTableDataManager.java| 48 - .../core/data/manager/InstanceDataManager.java | 14 -- .../pinot/core/data/manager/TableDataManager.java | 15 --- .../query/executor/ServerQueryExecutorV1Impl.java | 32 +- .../data/manager/BaseTableDataManagerTest.java | 5 --- .../pinot/query/executor/QueryExecutorTest.java| 50 ++ .../starter/helix/HelixInstanceDataManager.java| 16 --- .../SegmentOnlineOfflineStateModelFactory.java | 5 --- 9 files changed, 13 insertions(+), 174 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Revert "Set processingException when all queried segments cannot be acquired (#3942)"
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch revert
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 353c7b106179b00e004b3e976bc6068d9b01b950
Author: Jennifer Dai
AuthorDate: Mon Sep 23 13:36:40 2019 -0700
Revert "Set processingException when all queried segments cannot be
acquired (#3942)"
This reverts commit d4f2ecef660ab1d4efa9696a53b0623aac867c3f.
---
.../pinot/common/exception/QueryException.java | 2 -
.../core/data/manager/BaseTableDataManager.java| 48 -
.../core/data/manager/InstanceDataManager.java | 14 --
.../pinot/core/data/manager/TableDataManager.java | 15 ---
.../query/executor/ServerQueryExecutorV1Impl.java | 32 +-
.../data/manager/BaseTableDataManagerTest.java | 5 ---
.../pinot/query/executor/QueryExecutorTest.java| 50 ++
.../starter/helix/HelixInstanceDataManager.java| 16 ---
.../SegmentOnlineOfflineStateModelFactory.java | 5 ---
9 files changed, 13 insertions(+), 174 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
b/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
index 6f83c8d..b0b9be5 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/exception/QueryException.java
@@ -41,7 +41,6 @@ public class QueryException {
public static final int SEGMENT_PLAN_EXECUTION_ERROR_CODE = 160;
public static final int COMBINE_SEGMENT_PLAN_TIMEOUT_ERROR_CODE = 170;
public static final int ACCESS_DENIED_ERROR_CODE = 180;
- public static final int SEGMENTS_MISSING_ERROR_CODE = 190;
public static final int QUERY_EXECUTION_ERROR_CODE = 200;
// TODO: Handle these errors in broker
public static final int SERVER_SHUTTING_DOWN_ERROR_CODE = 210;
@@ -98,7 +97,6 @@ public class QueryException {
public static final ProcessingException QUERY_VALIDATION_ERROR = new
ProcessingException(QUERY_VALIDATION_ERROR_CODE);
public static final ProcessingException UNKNOWN_ERROR = new
ProcessingException(UNKNOWN_ERROR_CODE);
public static final ProcessingException QUOTA_EXCEEDED_ERROR = new
ProcessingException(TOO_MANY_REQUESTS_ERROR_CODE);
- public static final ProcessingException SEGMENTS_MISSING_ERROR = new
ProcessingException(SEGMENTS_MISSING_ERROR_CODE);
static {
JSON_PARSING_ERROR.setMessage("JsonParsingError");
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
index 3f074a1..a456691 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
@@ -19,13 +19,10 @@
package org.apache.pinot.core.data.manager;
import com.google.common.base.Preconditions;
-import com.google.common.cache.Cache;
-import com.google.common.cache.CacheBuilder;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.TimeUnit;
import javax.annotation.Nonnull;
import javax.annotation.concurrent.ThreadSafe;
import org.apache.helix.ZNRecord;
@@ -45,13 +42,9 @@ import org.slf4j.LoggerFactory;
@ThreadSafe
public abstract class BaseTableDataManager implements TableDataManager {
private static final Logger LOGGER =
LoggerFactory.getLogger(BaseTableDataManager.class);
- // cache deleted segment names for utmost this duration
- private static final int MAX_CACHE_DURATION_SEC = 6 * 3600; // 6 hours
protected final ConcurrentHashMap
_segmentDataManagerMap = new ConcurrentHashMap<>();
- protected Cache _deletedSegmentsCache;
-
protected TableDataManagerConfig _tableDataManagerConfig;
protected String _instanceId;
protected ZkHelixPropertyStore _propertyStore;
@@ -66,7 +59,6 @@ public abstract class BaseTableDataManager implements
TableDataManager {
@Nonnull ZkHelixPropertyStore propertyStore, @Nonnull
ServerMetrics serverMetrics) {
LOGGER.info("Initializing table data manager for table: {}",
tableDataManagerConfig.getTableName());
-_deletedSegmentsCache =
CacheBuilder.newBuilder().expireAfterWrite(MAX_CACHE_DURATION_SEC,
TimeUnit.SECONDS).build();
_tableDataManagerConfig = tableDataManagerConfig;
_instanceId = instanceId;
_propertyStore = propertyStore;
@@ -125,8 +117,6 @@ public abstract class BaseTableDataManager implements
TableDataManager {
ImmutableSegmentDataManager newSegmentManager = new
ImmutableSegmentDataManager(immutableSegment);
SegmentDataManager oldSegmentManager =
_segmentDataManagerMap.put(segmentName, newSegmentMan
[incubator-pinot] branch revert created (now 353c7b1)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch revert in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 353c7b1 Revert "Set processingException when all queried segments cannot be acquired (#3942)" This branch includes the following new commits: new 353c7b1 Revert "Set processingException when all queried segments cannot be acquired (#3942)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch proxy deleted (was ddb8c5f)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch proxy in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was ddb8c5f Adding a constructor to create the HttpClient with a proxy The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimetest updated (104825e -> 968b1eb)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimetest in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. omit 104825e Refactoring add 968b1eb Refactoring This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (104825e) \ N -- N -- N refs/heads/realtimetest (968b1eb) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../pinot/core/data/manager/realtime/SegmentCommitter.java | 13 + 1 file changed, 5 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch realtimetest updated: Refactoring
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch realtimetest
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/realtimetest by this push:
new 104825e Refactoring
104825e is described below
commit 104825edcc15c14330b5f140c64ef04b8af939ed
Author: Jennifer Dai
AuthorDate: Fri Sep 6 11:08:16 2019 -0700
Refactoring
---
.../realtime/LLRealtimeSegmentDataManager.java | 17 -
.../data/manager/realtime/SegmentCommitter.java| 84 --
2 files changed, 43 insertions(+), 58 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
index 35beae5..eb0c51b 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
@@ -768,9 +768,20 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
}
SegmentCompletionProtocol.Response returnedResponse;
boolean isSplitCommit = response.isSplitCommit() &&
_indexLoadingConfig.isEnableSplitCommit();
-SegmentCommitter segmentCommitter = new SegmentCommitter(isSplitCommit,
_segmentBuildDescriptor, _segmentNameStr,
-_currentOffset, _numRowsConsumed, _instanceId, _isOffHeap,
_protocolHandler, _memoryManager, _indexLoadingConfig, response);
-returnedResponse = segmentCommitter.commitSegment();
+
+SegmentCompletionProtocol.Request.Params params = new
SegmentCompletionProtocol.Request.Params();
+
+
params.withSegmentName(_segmentNameStr).withOffset(_currentOffset).withNumRows(_numRowsConsumed)
+
.withInstanceId(_instanceId).withBuildTimeMillis(_segmentBuildDescriptor.getBuildTimeMillis())
+.withSegmentSizeBytes(_segmentBuildDescriptor.getSegmentSizeBytes())
+.withWaitTimeMillis(_segmentBuildDescriptor.getWaitTimeMillis());
+
+if (_isOffHeap) {
+ params.withMemoryUsedBytes(_memoryManager.getTotalAllocatedBytes());
+}
+
+SegmentCommitter segmentCommitter = new SegmentCommitter(isSplitCommit,
_segmentNameStr, _protocolHandler, _indexLoadingConfig, response, params);
+returnedResponse = segmentCommitter.commitSegment(_segmentBuildDescriptor,
_currentOffset, _numRowsConsumed);
if
(!returnedResponse.getStatus().equals(SegmentCompletionProtocol.ControllerResponseStatus.COMMIT_SUCCESS))
{
return false;
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitter.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitter.java
index e0320d2..45d5ca2 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitter.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/SegmentCommitter.java
@@ -15,65 +15,45 @@ import org.slf4j.LoggerFactory;
public class SegmentCommitter {
private boolean _isSplitCommit;
- private LLRealtimeSegmentDataManager.SegmentBuildDescriptor
_segmentBuildDescriptor;
private String _segmentNameStr;
- private volatile long _currentOffset;
- private volatile int _numRowsConsumed;
- private String _instanceId;
- private boolean _isOffHeap;
private ServerSegmentCompletionProtocolHandler _protocolHandler;
- private PinotDataBufferMemoryManager _memoryManager;
private IndexLoadingConfig _indexLoadingConfig;
private SegmentCompletionProtocol.Response _prevResponse;
+ private SegmentCompletionProtocol.Request.Params _params;
private final Logger SEGMENT_LOGGER;
- public SegmentCommitter(boolean isSplitCommit,
LLRealtimeSegmentDataManager.SegmentBuildDescriptor segmentBuildDescriptor,
- String segmentNameStr, long currentOffset, int numRowsConsumed, String
instanceId, boolean isOffHeap,
- ServerSegmentCompletionProtocolHandler protocolHandler,
PinotDataBufferMemoryManager memoryManager,
- IndexLoadingConfig indexLoadingConfig,
SegmentCompletionProtocol.Response prevResponse) {
+ public SegmentCommitter(boolean isSplitCommit, String segmentNameStr,
ServerSegmentCompletionProtocolHandler protocolHandler,
+ IndexLoadingConfig indexLoadingConfig,
SegmentCompletionProtocol.Response prevResponse,
SegmentCompletionProtocol.Request.Params params) {
_isSplitCommit = isSplitCommit;
-_segmentBuildDescriptor = segmentBuildDescriptor;
_segmentNameStr = segmentNameStr;
-_currentOffset = currentOffset;
-_numRowsConsumed = numRowsConsumed;
-_instanceId = instanceId;
-_isOffHeap = isOffHeap;
_protocolHandler = protocolHandler;
-_memoryManager = memoryManager;
_indexLoadingConfig = indexLoadingConfig;
_prev
[incubator-pinot] branch deletepreprocessoutput created (now 24d551c)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch deletepreprocessoutput in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 24d551c Deleting preprocess output post segment creation This branch includes the following new commits: new 24d551c Deleting preprocess output post segment creation The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Deleting preprocess output post segment creation
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch deletepreprocessoutput
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 24d551c890322ecef1aaa75454ffe903c7a519f4
Author: Jennifer Dai
AuthorDate: Fri Sep 6 10:49:13 2019 -0700
Deleting preprocess output post segment creation
---
.../java/org/apache/pinot/hadoop/job/SegmentCreationJob.java | 10 ++
1 file changed, 10 insertions(+)
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
index aa68f10..5cba226 100644
---
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
+++
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
@@ -63,6 +63,7 @@ public class SegmentCreationJob extends BaseSegmentJob {
protected final Path _schemaFile;
protected final String _defaultPermissionsMask;
protected final List _pushLocations;
+ protected Path _preprocessOutputPath = null;
protected FileSystem _fileSystem;
@@ -90,6 +91,11 @@ public class SegmentCreationJob extends BaseSegmentJob {
_pushLocations = null;
}
+String preprocessOutputPath =
_properties.getProperty(JobConfigConstants.PREPROCESS_PATH_TO_OUTPUT);
+if (preprocessOutputPath != null) {
+ _preprocessOutputPath = getPathFromProperty(preprocessOutputPath);
+}
+
_logger.info("*");
_logger.info("Input Pattern: {}", _inputPattern);
_logger.info("Output Directory: {}", _outputDir);
@@ -192,6 +198,10 @@ public class SegmentCreationJob extends BaseSegmentJob {
// Delete the staging directory
_logger.info("Deleting the staging directory: {}", _stagingDir);
_fileSystem.delete(_stagingDir, true);
+
+if (_preprocessOutputPath != null) {
+ _fileSystem.delete(_preprocessOutputPath, true);
+}
}
@Nullable
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch paths updated (273658f -> fa31be6)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch paths in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 273658f Disallowing multiple inputs in preprocess add fa31be6 Disallowing multiple inputs in preprocess This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (273658f) \ N -- N -- N refs/heads/paths (fa31be6) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch paths updated (9a39881 -> 273658f)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch paths in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 9a39881 Disallowing multiple inputs in preprocess add 273658f Disallowing multiple inputs in preprocess This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (9a39881) \ N -- N -- N refs/heads/paths (273658f) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Disallowing multiple inputs in preprocess
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch paths
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 9a3988198ab9de3fc1bf01f215a6f321a0475fb6
Author: Jennifer Dai
AuthorDate: Fri Sep 6 10:42:03 2019 -0700
Disallowing multiple inputs in preprocess
---
.../pinot/hadoop/job/SegmentPreprocessingJob.java | 21 +++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
index 942fb6d..3d07245 100644
---
a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
+++
b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java
@@ -83,6 +83,7 @@ public class SegmentPreprocessingJob extends BaseSegmentJob {
private String _partitionFunction;
private String _sortedColumn;
private int _numOutputFiles;
+ private boolean _isMultipleInput = false;
private final Path _inputSegmentDir;
private final Path _preprocessedOutputDir;
@@ -101,8 +102,19 @@ public class SegmentPreprocessingJob extends
BaseSegmentJob {
_enablePreprocessing =
Boolean.parseBoolean(_properties.getProperty(JobConfigConstants.ENABLE_PREPROCESSING));
-// get input/output paths.
-_inputSegmentDir =
Preconditions.checkNotNull(getPathFromProperty(JobConfigConstants.PATH_TO_INPUT));
+String inputPath =
Preconditions.checkNotNull(properties.getProperty(JobConfigConstants.PATH_TO_INPUT));
+
+// We cannot support this because mapreduce takes complete control of the
output path. In order to support this, we
+// would need control to pipe the exact folders we receive as input to
multiple outputs. While we can programmatically
+// determine record by record what goes into each output path, this does
not support our use case. Each folder is a
+// separate "day," and frequently, our customers will have two dates in
one file, due to timezone of data, so we
+// are not able to distinguish what is "today's" vs. "tomorrow's" data by
solely looking at the record.
+if (inputPath.split(",").length > 1) {
+ _isMultipleInput = true;
+}
+
+// get input path/output paths.
+_inputSegmentDir = getPathFromProperty(JobConfigConstants.PATH_TO_INPUT);
_preprocessedOutputDir =
getPathFromProperty(JobConfigConstants.PREPROCESS_PATH_TO_OUTPUT);
_rawTableName =
Preconditions.checkNotNull(_properties.getProperty(JobConfigConstants.SEGMENT_TABLE_NAME));
@@ -138,6 +150,11 @@ public class SegmentPreprocessingJob extends
BaseSegmentJob {
_logger.info("Starting {}", getClass().getSimpleName());
}
+if (_isMultipleInput) {
+ _logger.info("Skipping pre-processing, multiple inputs detected. Not
supported");
+ return;
+}
+
_fileSystem = FileSystem.get(_conf);
final List inputDataPaths = getDataFilePaths(_inputSegmentDir);
Preconditions.checkState(inputDataPaths.size() != 0, "No files in the
input directory.");
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch paths created (now 9a39881)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch paths in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 9a39881 Disallowing multiple inputs in preprocess This branch includes the following new commits: new 9a39881 Disallowing multiple inputs in preprocess The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (76e0c62 -> 68092ab)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 76e0c62 Adding Predicate supports for Bytes Column without dictionary (#4586) add 68092ab Adding integration test for Hadoop pre-process job (#4577) No new revisions were added by this update. Summary of changes: .../name/NormalizedDateSegmentNameGenerator.java | 4 - .../pinot/hadoop/job/SegmentCreationJob.java | 17 +--- .../pinot/hadoop/job/SegmentPreprocessingJob.java | 14 +--- .../job/mappers/SegmentPreprocessingMapper.java| 17 ++-- .../pinot/hadoop/utils/JobPreparationHelper.java | 11 +++ .../pinot/integration/tests/ClusterTest.java | 8 +- ...mentBuildPushOfflineClusterIntegrationTest.java | 97 +- 7 files changed, 86 insertions(+), 82 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch cleanup deleted (was 084edf3)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch cleanup in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. was 084edf3 Fixing error The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch cleanup updated (c9a26fb -> 084edf3)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch cleanup in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from c9a26fb Adding umask add 084edf3 Fixing error No new revisions were added by this update. Summary of changes: .../java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java | 8 1 file changed, 4 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Refactoring realtime segment data manager
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch realtimetest
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 04bdd56125c920c44092001284874e2d04e18191
Author: Jennifer Dai
AuthorDate: Wed Sep 4 15:01:34 2019 -0700
Refactoring realtime segment data manager
---
.../realtime/LLRealtimeSegmentDataManager.java | 82 +---
.../data/manager/realtime/SegmentCommitter.java| 137 +
2 files changed, 141 insertions(+), 78 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
index 9c254d9..35beae5 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
@@ -760,57 +760,6 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
}
}
- private SegmentCompletionProtocol.Response
doSplitCommit(SegmentCompletionProtocol.Response prevResponse) {
-final File segmentTarFile = new
File(_segmentBuildDescriptor.getSegmentTarFilePath());
-SegmentCompletionProtocol.Request.Params params = new
SegmentCompletionProtocol.Request.Params();
-
-
params.withSegmentName(_segmentNameStr).withOffset(_currentOffset).withNumRows(_numRowsConsumed)
-
.withInstanceId(_instanceId).withBuildTimeMillis(_segmentBuildDescriptor.getBuildTimeMillis())
-.withSegmentSizeBytes(_segmentBuildDescriptor.getSegmentSizeBytes())
-.withWaitTimeMillis(_segmentBuildDescriptor.getWaitTimeMillis());
-if (_isOffHeap) {
- params.withMemoryUsedBytes(_memoryManager.getTotalAllocatedBytes());
-}
-SegmentCompletionProtocol.Response segmentCommitStartResponse =
_protocolHandler.segmentCommitStart(params);
-if (!segmentCommitStartResponse.getStatus()
-
.equals(SegmentCompletionProtocol.ControllerResponseStatus.COMMIT_CONTINUE)) {
- segmentLogger.warn("CommitStart failed with response {}",
segmentCommitStartResponse.toJsonString());
- return SegmentCompletionProtocol.RESP_FAILED;
-}
-
-params = new SegmentCompletionProtocol.Request.Params();
-
params.withOffset(_currentOffset).withSegmentName(_segmentNameStr).withInstanceId(_instanceId);
-SegmentCompletionProtocol.Response segmentCommitUploadResponse =
-_protocolHandler.segmentCommitUpload(params, segmentTarFile,
prevResponse.getControllerVipUrl());
-if (!segmentCommitUploadResponse.getStatus()
-
.equals(SegmentCompletionProtocol.ControllerResponseStatus.UPLOAD_SUCCESS)) {
- segmentLogger.warn("Segment upload failed with response {}",
segmentCommitUploadResponse.toJsonString());
- return SegmentCompletionProtocol.RESP_FAILED;
-}
-
-params = new SegmentCompletionProtocol.Request.Params();
-
params.withInstanceId(_instanceId).withOffset(_currentOffset).withSegmentName(_segmentNameStr)
-
.withSegmentLocation(segmentCommitUploadResponse.getSegmentLocation()).withNumRows(_numRowsConsumed)
-.withBuildTimeMillis(_segmentBuildDescriptor.getBuildTimeMillis())
-.withSegmentSizeBytes(_segmentBuildDescriptor.getSegmentSizeBytes())
-.withWaitTimeMillis(_segmentBuildDescriptor.getWaitTimeMillis());
-if (_isOffHeap) {
- params.withMemoryUsedBytes(_memoryManager.getTotalAllocatedBytes());
-}
-SegmentCompletionProtocol.Response commitEndResponse;
-if (_indexLoadingConfig.isEnableSplitCommitEndWithMetadata()) {
- commitEndResponse =
_protocolHandler.segmentCommitEndWithMetadata(params,
_segmentBuildDescriptor.getMetadataFiles());
-} else {
- commitEndResponse = _protocolHandler.segmentCommitEnd(params);
-}
-
-if
(!commitEndResponse.getStatus().equals(SegmentCompletionProtocol.ControllerResponseStatus.COMMIT_SUCCESS))
{
- segmentLogger.warn("CommitEnd failed with response {}",
commitEndResponse.toJsonString());
- return SegmentCompletionProtocol.RESP_FAILED;
-}
-return commitEndResponse;
- }
-
protected boolean commitSegment(SegmentCompletionProtocol.Response response)
{
final String segTarFileName =
_segmentBuildDescriptor.getSegmentTarFilePath();
File segTarFile = new File(segTarFileName);
@@ -818,15 +767,10 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
throw new RuntimeException("Segment file does not exist:" +
segTarFileName);
}
SegmentCompletionProtocol.Response returnedResponse;
-if (response.isSplitCommit() && _indexLoadingConfig.isEnableSplitCommit())
{
- // Send segmentStart, segmentUpload, & segmentCommitEnd to the controller
- // if that
[incubator-pinot] branch realtimetest created (now 04bdd56)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch realtimetest in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 04bdd56 Refactoring realtime segment data manager This branch includes the following new commits: new 04bdd56 Refactoring realtime segment data manager The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch cleanup updated (e9b0ea0 -> 2436154)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch cleanup in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from e9b0ea0 Addressing comments add 2436154 Addressing comments No new revisions were added by this update. Summary of changes: .../java/org/apache/pinot/hadoop/job/SegmentPreprocessingJob.java| 4 .../apache/pinot/hadoop/job/mappers/SegmentPreprocessingMapper.java | 5 ++--- 2 files changed, 2 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch cleanup updated (a8ab052 -> e9b0ea0)
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a change to branch cleanup
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard a8ab052 Fixing test
discard aff5441 Adding preprocess test
add 7c7b6cc Support transform functions with AVG aggregation function
(#4557)
add 9b5775e [Issue #4551] Remove memory allocation for virtual columns in
consumi… (#4555)
add d8d0bb5 Add metrics on Helix and Pinot controller leadership (#4561)
add 38c73aa Fix the issue of scheduling task on all tables (#4563)
add 23ffd67 Handle the virtual column logic in FieldSpec instead of
Schema (#4562)
add c2f6b25 Indexed table (#4547)
add 9edb2cc [TE] frontend - harleyjj/alert-details - add rule and
modifier columns to anomaly table (#4564)
add 77446e7 [TE] support entity group expression (#4377)
add e98efcb [TE] fix multi-day granularity alignment issue (#4566)
add 9180dac fixed "StartTree" typo (#4571)
add 16cd4fa fixed hyperlink to PinotFS (#4570)
add c30ef17 [TE] only store evaluations for top dimensions (#4572)
add 3216159 Support aws s3 with Parquet in pinot-tools (#4556)
add fbc28d8 [TE] Add timeout for all tasks and clean up RUNNING states if
timeout (#4554)
add 9df877d Use docker compose for pinot quickstart (#4575)
add 584368d Merge LLC related integration tests (#4573)
add f3aa4db Retrieve routing table by exact table name instead of by
prefix (#3037)
add a1c5af7 [TE] Return the anomaly details to render snapshot (#4576)
add 8712a59 Adding preprocess test
add fa00ab3 Fixing test
add e9b0ea0 Addressing comments
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a8ab052)
\
N -- N -- N refs/heads/cleanup (e9b0ea0)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
docker/Dockerfile | 5 +-
docker/README.md | 33 ++-
docker/docker-build.sh | 11 +-
docker/docker-compose.yml | 82 ++
docs/architecture.rst | 2 +-
docs/pluggable_storage.rst | 2 +-
.../routing/HelixExternalViewBasedRouting.java | 4 +-
.../org/apache/pinot/common/data/FieldSpec.java| 11 +
.../java/org/apache/pinot/common/data/Schema.java | 17 +-
.../pinot/common/metrics/ControllerGauge.java | 6 +
.../pinot/controller/LeadControllerManager.java| 10 +
.../helix/core/minion/PinotTaskManager.java| 11 +-
.../core/periodictask/ControllerPeriodicTask.java | 14 +-
.../pinot/controller/helix/ControllerTest.java | 2 +-
.../apache/pinot/core/data/order/OrderByUtils.java | 295 +
.../apache/pinot/core/data/readers/FileFormat.java | 2 +-
.../core/data/table/ConcurrentIndexedTable.java| 137 ++
.../apache/pinot/core/data/table/IndexedTable.java | 69 +
.../java/org/apache/pinot/core/data/table/Key.java | 49 ++--
.../{plan/Plan.java => data/table/Record.java} | 28 +-
.../pinot/core/data/table/SimpleIndexedTable.java | 126 +
.../org/apache/pinot/core/data/table/Table.java| 68 +
.../pinot/core/indexsegment/IndexSegment.java | 2 +-
.../generator/SegmentGeneratorConfig.java | 8 +-
.../immutable/ImmutableSegmentImpl.java| 27 +-
.../immutable/ImmutableSegmentLoader.java | 6 +-
.../indexsegment/mutable/MutableSegmentImpl.java | 117 +---
.../operator/docvalsets/TransformBlockValSet.java | 6 +
.../creator/impl/SegmentColumnarIndexCreator.java | 5 +-
.../impl/SegmentIndexCreationDriverImpl.java | 20 +-
.../VirtualColumnProviderFactory.java | 6 +-
.../pinot/core/data/order/OrderByUtilsTest.java| 265 ++
.../pinot/core/data/table/IndexedTableTest.java| 275 +++
.../MutableSegmentImplAggregateMetricsTest.java| 30 ++-
.../apache/pinot/queries/TransformQueriesTest.java | 267 +++
.../pinot/hadoop/job/SegmentPreprocessingJob.java | 2 +-
.../job/mappers/SegmentPreprocessingMapper.java| 26 +-
.../tests/LLCRealtimeClusterIntegrationTest.java | 100 ---
...CRealtimeClusterSpli
[incubator-pinot] branch proxy created (now ddb8c5f)
This is an automated email from the ASF dual-hosted git repository. jenniferdai pushed a change to branch proxy in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at ddb8c5f Adding a constructor to create the HttpClient with a proxy This branch includes the following new commits: new ddb8c5f Adding a constructor to create the HttpClient with a proxy The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
