(pinot) branch mark-resize-as-NumGroupsLimitReached updated (61cb64f839 -> cc9aaa8e18)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch mark-resize-as-NumGroupsLimitReached
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 61cb64f839 Introduce isBrokerResized flag in broker reduce phase
 add cc9aaa8e18 Introduce isBrokerResized flag in broker reduce phase

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (61cb64f839)
\
 N -- N -- N   refs/heads/mark-resize-as-NumGroupsLimitReached 
(cc9aaa8e18)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../pinot/common/response/broker/BrokerResponseNative.java |  2 +-
 .../common/response/broker/BrokerResponseNativeV2.java | 14 --
 .../pinot/query/runtime/operator/AggregateOperator.java|  3 ++-
 .../pinot/query/runtime/operator/MultiStageOperator.java   |  1 +
 .../query/runtime/operator/AggregateOperatorTest.java  |  2 ++
 5 files changed, 18 insertions(+), 4 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch mark-resize-as-NumGroupsLimitReached updated (2c749dfc59 -> 61cb64f839)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch mark-resize-as-NumGroupsLimitReached
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 2c749dfc59 Mark resize as NumGroupsLimitReached in broker reduce phase
 add 61cb64f839 Introduce isBrokerResized flag in broker reduce phase

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (2c749dfc59)
\
 N -- N -- N   refs/heads/mark-resize-as-NumGroupsLimitReached 
(61cb64f839)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../requesthandler/BaseSingleStageBrokerRequestHandler.java|  5 +
 .../main/java/org/apache/pinot/common/metrics/BrokerMeter.java |  6 --
 .../java/org/apache/pinot/common/response/BrokerResponse.java  |  5 +
 .../pinot/common/response/broker/BrokerResponseNative.java | 10 ++
 .../pinot/core/query/reduce/GroupByDataTableReducer.java   |  5 ++---
 5 files changed, 26 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Mark resize as NumGroupsLimitReached in broker reduce phase

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch mark-resize-as-NumGroupsLimitReached
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 2c749dfc5945c9b27b3acdc3e6b50026c61a06a0
Author: jlli_LinkedIn 
AuthorDate: Fri May 24 17:35:53 2024 -0700

Mark resize as NumGroupsLimitReached in broker reduce phase
---
 .../main/java/org/apache/pinot/core/data/table/IndexedTable.java   | 7 ++-
 .../main/java/org/apache/pinot/core/data/table/TableResizer.java   | 5 +++--
 .../apache/pinot/core/query/reduce/GroupByDataTableReducer.java| 3 +++
 3 files changed, 12 insertions(+), 3 deletions(-)

diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/data/table/IndexedTable.java 
b/pinot-core/src/main/java/org/apache/pinot/core/data/table/IndexedTable.java
index 012fdc1170..5dec9a78f3 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/data/table/IndexedTable.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/data/table/IndexedTable.java
@@ -48,6 +48,7 @@ public abstract class IndexedTable extends BaseTable {
 
   protected Collection _topRecords;
   private int _numResizes;
+  private boolean _isResized = false;
   private long _resizeTimeNs;
 
   /**
@@ -138,7 +139,7 @@ public abstract class IndexedTable extends BaseTable {
   protected void resize() {
 assert _hasOrderBy;
 long startTimeNs = System.nanoTime();
-_tableResizer.resizeRecordsMap(_lookupMap, _trimSize);
+_isResized |= _tableResizer.resizeRecordsMap(_lookupMap, _trimSize);
 long resizeTimeNs = System.nanoTime() - startTimeNs;
 _numResizes++;
 _resizeTimeNs += resizeTimeNs;
@@ -186,6 +187,10 @@ public abstract class IndexedTable extends BaseTable {
 return _numResizes;
   }
 
+  public boolean isResized() {
+return _isResized;
+  }
+
   public long getResizeTimeMs() {
 return TimeUnit.NANOSECONDS.toMillis(_resizeTimeNs);
   }
diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/data/table/TableResizer.java 
b/pinot-core/src/main/java/org/apache/pinot/core/data/table/TableResizer.java
index 4299e5665e..b9acabab36 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/data/table/TableResizer.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/data/table/TableResizer.java
@@ -174,10 +174,10 @@ public class TableResizer {
   /**
* Resizes the recordsMap to the given size.
*/
-  public void resizeRecordsMap(Map recordsMap, int size) {
+  public boolean resizeRecordsMap(Map recordsMap, int size) {
 int numRecordsToEvict = recordsMap.size() - size;
 if (numRecordsToEvict <= 0) {
-  return;
+  return false;
 }
 if (numRecordsToEvict <= size) {
   // Fewer records to evict than retain, make a heap of records to evict
@@ -195,6 +195,7 @@ public class TableResizer {
 recordsMap.put(recordToRetain._key, recordToRetain._record);
   }
 }
+return true;
   }
 
   /**
diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GroupByDataTableReducer.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GroupByDataTableReducer.java
index c0a109f7e4..4e76b9c4f1 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GroupByDataTableReducer.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/GroupByDataTableReducer.java
@@ -148,6 +148,9 @@ public class GroupByDataTableReducer implements 
DataTableReducer {
   brokerMetrics.addMeteredTableValue(rawTableName, 
BrokerMeter.NUM_RESIZES, indexedTable.getNumResizes());
   brokerMetrics.addValueToTableGauge(rawTableName, 
BrokerGauge.RESIZE_TIME_MS, indexedTable.getResizeTimeMs());
 }
+// Take resizing in broker as one of the factors for NumGroupsLimitReached
+brokerResponseNative.setNumGroupsLimitReached(
+brokerResponseNative.isNumGroupsLimitReached() || 
indexedTable.isResized());
 int numRecords = indexedTable.size();
 Iterator sortedIterator = indexedTable.iterator();
 


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch mark-resize-as-NumGroupsLimitReached created (now 2c749dfc59)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch mark-resize-as-NumGroupsLimitReached
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 2c749dfc59 Mark resize as NumGroupsLimitReached in broker reduce phase

This branch includes the following new commits:

 new 2c749dfc59 Mark resize as NumGroupsLimitReached in broker reduce phase

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch fix-broker-metrics deleted (was ca7004a14b)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-broker-metrics
in repository https://gitbox.apache.org/repos/asf/pinot.git


 was ca7004a14b Fetch query quota capacity utilization rate metric in a 
callback function

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch master updated (5440c6e443 -> 984b561669)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git


from 5440c6e443 Bugfix: Validate minionInstanceTag during task-generation 
(#13092)
 add 984b561669 Fetch query quota capacity utilization rate metric in a 
callback function (#12767)

No new revisions were added by this update.

Summary of changes:
 .../HelixExternalViewBasedQueryQuotaManager.java   | 30 +-
 .../apache/pinot/broker/queryquota/HitCounter.java | 12 -
 .../pinot/broker/queryquota/MaxHitRateTracker.java | 12 +
 .../broker/queryquota/MaxHitRateTrackerTest.java   |  5 
 4 files changed, 51 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch master updated (67cb52c04b -> 1d807df401)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git


from 67cb52c04b Refine SegmentFetcherFactory (#12936)
 add 1d807df401 Add validation check for forward index disabled if it's a 
REALTIME table (#12838)

No new revisions were added by this update.

Summary of changes:
 .../segment/local/utils/TableConfigUtils.java  | 20 ---
 .../segment/local/utils/TableConfigUtilsTest.java  | 23 --
 2 files changed, 34 insertions(+), 9 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation deleted (was 7dfd5c619e)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch 
add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation
in repository https://gitbox.apache.org/repos/asf/pinot.git


 was 7dfd5c619e Add validation check for forward index disabled if it's a 
REALTIME table

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation updated (75498ba651 -> 7dfd5c619e)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch 
add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 75498ba651 Add validation check for forward index disabled if it's a 
REALTIME table
 add 7dfd5c619e Add validation check for forward index disabled if it's a 
REALTIME table

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (75498ba651)
\
 N -- N -- N   
refs/heads/add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation
 (7dfd5c619e)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../org/apache/pinot/segment/local/utils/TableConfigUtils.java   | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch clean-up-task-manager-for-tables deleted (was 7068a246e9)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch clean-up-task-manager-for-tables
in repository https://gitbox.apache.org/repos/asf/pinot.git


 was 7068a246e9 Do task related cleanup for non-controller leader

This change permanently discards the following revisions:

 discard 7068a246e9 Do task related cleanup for non-controller leader


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Do task related cleanup for non-controller leader

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch clean-up-task-manager-for-tables
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 7068a246e9851329a7580cbba4479b8599aa64a2
Author: jlli_LinkedIn 
AuthorDate: Mon Apr 15 15:39:13 2024 -0700

Do task related cleanup for non-controller leader
---
 .../controller/helix/core/minion/PinotTaskManager.java   | 16 +++-
 .../helix/core/minion/generator/PinotTaskGenerator.java  |  7 +++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java
index 2cdbf8c1df..a96e550738 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java
@@ -693,11 +693,25 @@ public class PinotTaskManager extends 
ControllerPeriodicTask {
 scheduleTasks(tableNamesWithType, true);
   }
 
+  @Override
+  protected void nonLeaderCleanup(List tableNamesWithType) {
+LOGGER.info("Cleaning up all task generators for tables: " + 
tableNamesWithType);
+for (String taskType : _taskGeneratorRegistry.getAllTaskTypes()) {
+  PinotTaskGenerator pinotTaskGenerator = 
_taskGeneratorRegistry.getTaskGenerator(taskType);
+  if (pinotTaskGenerator != null) {
+pinotTaskGenerator.nonLeaderCleanUp(tableNamesWithType);
+  }
+}
+  }
+
   @Override
   public void cleanUpTask() {
 LOGGER.info("Cleaning up all task generators");
 for (String taskType : _taskGeneratorRegistry.getAllTaskTypes()) {
-  _taskGeneratorRegistry.getTaskGenerator(taskType).nonLeaderCleanUp();
+  PinotTaskGenerator pinotTaskGenerator = 
_taskGeneratorRegistry.getTaskGenerator(taskType);
+  if (pinotTaskGenerator != null) {
+pinotTaskGenerator.nonLeaderCleanUp();
+  }
 }
   }
 
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/PinotTaskGenerator.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/PinotTaskGenerator.java
index ad42135886..fe98e971f8 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/PinotTaskGenerator.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/PinotTaskGenerator.java
@@ -86,6 +86,13 @@ public interface PinotTaskGenerator {
   default void nonLeaderCleanUp() {
   }
 
+  /**
+   * Given the list of table names, performs necessary cleanups (e.g. remove 
metrics) when the controller leadership
+   * changes.
+   */
+  default void nonLeaderCleanUp(List tableNameWithType) {
+  }
+
   /**
* Gets the minionInstanceTag for the tableConfig
*/


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch clean-up-task-manager-for-tables created (now 7068a246e9)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch clean-up-task-manager-for-tables
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 7068a246e9 Do task related cleanup for non-controller leader

This branch includes the following new commits:

 new 7068a246e9 Do task related cleanup for non-controller leader

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch unify-broker-empty-result deleted (was 56c7a9ec8d)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch unify-broker-empty-result
in repository https://gitbox.apache.org/repos/asf/pinot.git


 was 56c7a9ec8d Unify the usage of broker empty results

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch unify-broker-empty-result updated (0b11c2964c -> 56c7a9ec8d)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch unify-broker-empty-result
in repository https://gitbox.apache.org/repos/asf/pinot.git


omit 0b11c2964c Unify the usage of broker empty results
 add 56c7a9ec8d Unify the usage of broker empty results

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (0b11c2964c)
\
 N -- N -- N   refs/heads/unify-broker-empty-result (56c7a9ec8d)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../test/java/org/apache/pinot/request/BrokerResponseNativeTest.java| 2 --
 1 file changed, 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch unify-broker-empty-result updated (8e5f7b8e41 -> 0b11c2964c)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch unify-broker-empty-result
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 8e5f7b8e41 Unify the usage of broker empty results
 add 0b11c2964c Unify the usage of broker empty results

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (8e5f7b8e41)
\
 N -- N -- N   refs/heads/unify-broker-empty-result (0b11c2964c)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/pinot/request/BrokerResponseNativeTest.java | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch unify-broker-empty-result created (now 8e5f7b8e41)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch unify-broker-empty-result
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 8e5f7b8e41 Unify the usage of broker empty results

This branch includes the following new commits:

 new 8e5f7b8e41 Unify the usage of broker empty results

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Unify the usage of broker empty results

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch unify-broker-empty-result
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 8e5f7b8e41dfbb65dfb26a0d6632215bdf52b521
Author: jlli_LinkedIn 
AuthorDate: Thu Apr 11 13:12:26 2024 -0700

Unify the usage of broker empty results
---
 .../apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java  | 2 +-
 .../pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java   | 2 +-
 .../org/apache/pinot/common/response/broker/BrokerResponseNative.java | 4 ++--
 .../java/org/apache/pinot/core/query/reduce/BrokerReduceService.java  | 2 +-
 .../apache/pinot/core/query/reduce/SelectionOnlyStreamingReducer.java | 2 +-
 .../org/apache/pinot/core/query/reduce/StreamingReduceService.java| 2 +-
 6 files changed, 7 insertions(+), 7 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
index b2d4e24d3f..b7b1e7d215 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
@@ -827,7 +827,7 @@ public abstract class BaseBrokerRequestHandler implements 
BrokerRequestHandler {
 }
 
 // Send empty response since we don't need to evaluate either offline or 
realtime request.
-BrokerResponseNative brokerResponse = BrokerResponseNative.empty();
+BrokerResponseNative brokerResponse = BrokerResponseNative.EMPTY_RESULT;
 // Extract source info from incoming request
 _queryLogger.log(
 new QueryLogger.QueryLogParams(requestId, query, requestContext, 
tableName, 0, null, brokerResponse,
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
index 01e4884d6a..b8015a9203 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
@@ -320,7 +320,7 @@ public class MultiStageBrokerRequestHandler extends 
BaseBrokerRequestHandler {
   }
 
   private BrokerResponseNative constructMultistageExplainPlan(String sql, 
String plan) {
-BrokerResponseNative brokerResponse = BrokerResponseNative.empty();
+BrokerResponseNative brokerResponse = BrokerResponseNative.EMPTY_RESULT;
 List rows = new ArrayList<>();
 rows.add(new Object[]{sql, plan});
 DataSchema multistageExplainResultSchema = new DataSchema(new 
String[]{"SQL", "PLAN"},
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
 
b/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
index 9fe098e26d..142e497ad0 100644
--- 
a/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNative.java
@@ -117,7 +117,7 @@ public class BrokerResponseNative implements BrokerResponse 
{
 
   /** Generate EXPLAIN PLAN output when queries are evaluated by Broker 
without going to the Server. */
   private static BrokerResponseNative getBrokerResponseExplainPlanOutput() {
-BrokerResponseNative brokerResponse = BrokerResponseNative.empty();
+BrokerResponseNative brokerResponse = BrokerResponseNative.EMPTY_RESULT;
 List rows = new ArrayList<>();
 rows.add(new Object[]{"BROKER_EVALUATE", 0, -1});
 brokerResponse.setResultTable(new 
ResultTable(DataSchema.EXPLAIN_RESULT_SCHEMA, rows));
@@ -127,7 +127,7 @@ public class BrokerResponseNative implements BrokerResponse 
{
   /**
* Get a new empty {@link BrokerResponseNative}.
*/
-  public static BrokerResponseNative empty() {
+  private static BrokerResponseNative empty() {
 return new BrokerResponseNative();
   }
 
diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
index c664a78e67..9e8d93940e 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/query/reduce/BrokerReduceService.java
@@ -63,7 +63,7 @@ public class BrokerReduceService extends BaseReduceService {
   Map dataTableMap, long 
reduceTimeOutMs, @Nullable BrokerMetrics brokerMetrics) {
 if (dataTableMap.isEmpty()) {
   // Empty response.
-  return BrokerResponseNative.empty();
+  return BrokerResponseNative.EMPTY_RESULT;
 }
 
 Map q

(pinot) branch add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation updated (c7f5887d55 -> 75498ba651)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch 
add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard c7f5887d55 Add validation check for forward index disabled when 
enabling columnar segment creation
 add 75498ba651 Add validation check for forward index disabled if it's a 
REALTIME table

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (c7f5887d55)
\
 N -- N -- N   
refs/heads/add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation
 (75498ba651)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../segment/local/utils/TableConfigUtils.java  | 19 +++---
 .../segment/local/utils/TableConfigUtilsTest.java  | 29 ++
 2 files changed, 22 insertions(+), 26 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation updated (4c2fca142d -> c7f5887d55)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch 
add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 4c2fca142d Add validation check for forward index disabled when 
enabling columnar segment creation
 add c7f5887d55 Add validation check for forward index disabled when 
enabling columnar segment creation

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (4c2fca142d)
\
 N -- N -- N   
refs/heads/add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation
 (c7f5887d55)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../segment/local/utils/TableConfigUtilsTest.java  | 40 +-
 1 file changed, 31 insertions(+), 9 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Add validation check for forward index disabled when enabling columnar segment creation

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch 
add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 4c2fca142da5600de9a98d6583a28729c7c6aa21
Author: jlli_LinkedIn 
AuthorDate: Mon Apr 8 17:12:08 2024 -0700

Add validation check for forward index disabled when enabling columnar 
segment creation
---
 .../org/apache/pinot/segment/local/utils/TableConfigUtils.java | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git 
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java
 
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java
index 8d31f5d399..fc50b56aa9 100644
--- 
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java
+++ 
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java
@@ -1310,11 +1310,11 @@ public final class TableConfigUtils {
 if (indexingConfig.getRangeIndexColumns() != null && 
indexingConfig.getRangeIndexColumns().contains(columnName)) {
   Preconditions.checkState(fieldSpec.isSingleValueField(), 
String.format("Feature not supported for multi-value "
   + "columns with range index. Cannot disable forward index for column 
%s. Disable range index on this "
-  + "column to use this feature", columnName));
+  + "column to use this feature.", columnName));
   Preconditions.checkState(indexingConfig.getRangeIndexVersion() == 
BitSlicedRangeIndexCreator.VERSION,
   String.format("Feature not supported for single-value columns with 
range index version < 2. Cannot disable "
   + "forward index for column %s. Either disable range index or 
create range index with"
-  + " version >= 2 to use this feature", columnName));
+  + " version >= 2 to use this feature.", columnName));
 }
 
 Preconditions.checkState(!indexingConfig.isOptimizeDictionaryForMetrics() 
&& !indexingConfig.isOptimizeDictionary(),
@@ -1334,6 +1334,12 @@ public final class TableConfigUtils {
   columnName,
   hasDictionary ? "enabled" : "disabled", hasInvertedIndex ? "enabled" 
: "disabled");
 }
+
+// For tables with columnMajorSegmentBuilderEnabled being true, the 
forward index should not be disabled.
+
Preconditions.checkState(!indexingConfig.isColumnMajorSegmentBuilderEnabled(), 
String.format(
+"Columnar segment generation is enabled. Cannot disable forward index 
for column %s. Disable columnar segment"
++ " generation to use this feature.",
+columnName));
   }
 
   private static void sanitize(TableConfig tableConfig) {


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation created (now 4c2fca142d)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch 
add-validation-check-for-forward-index-disabled-when-enabling-columnar-segment-creation
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 4c2fca142d Add validation check for forward index disabled when 
enabling columnar segment creation

This branch includes the following new commits:

 new 4c2fca142d Add validation check for forward index disabled when 
enabling columnar segment creation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch fix-broker-metrics created (now ca7004a14b)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-broker-metrics
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at ca7004a14b Fetch query quota capacity utilization rate metric in a 
callback function

This branch includes the following new commits:

 new ca7004a14b Fetch query quota capacity utilization rate metric in a 
callback function

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Fetch query quota capacity utilization rate metric in a callback function

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch fix-broker-metrics
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit ca7004a14bf049bcd8bb58a7f52bd80bbfab778c
Author: jlli_LinkedIn 
AuthorDate: Mon Apr 1 11:14:30 2024 -0700

Fetch query quota capacity utilization rate metric in a callback function
---
 .../HelixExternalViewBasedQueryQuotaManager.java   | 30 +-
 .../apache/pinot/broker/queryquota/HitCounter.java | 12 -
 .../pinot/broker/queryquota/MaxHitRateTracker.java | 12 +
 .../broker/queryquota/MaxHitRateTrackerTest.java   |  5 
 4 files changed, 51 insertions(+), 8 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java
index 04db0f6a42..dabb95867b 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java
@@ -224,6 +224,7 @@ public class HelixExternalViewBasedQueryQuotaManager 
implements ClusterChangeHan
   tableNameWithType, overallRate, previousRate, perBrokerRate, 
onlineCount, stat.getVersion());
 }
 addMaxBurstQPSCallbackTableGaugeIfNeeded(tableNameWithType, 
queryQuotaEntity);
+addQueryQuotaCapacityUtilizationRateTableGaugeIfNeeded(tableNameWithType, 
queryQuotaEntity);
 if (isQueryRateLimitDisabled()) {
   LOGGER.info("Query rate limiting is currently disabled for this broker. 
So it won't take effect immediately.");
 }
@@ -245,6 +246,7 @@ public class HelixExternalViewBasedQueryQuotaManager 
implements ClusterChangeHan
   queryQuotaEntity.setRateLimiter(null);
 }
 addMaxBurstQPSCallbackTableGaugeIfNeeded(tableNameWithType, 
queryQuotaEntity);
+addQueryQuotaCapacityUtilizationRateTableGaugeIfNeeded(tableNameWithType, 
queryQuotaEntity);
   }
 
   /**
@@ -256,6 +258,27 @@ public class HelixExternalViewBasedQueryQuotaManager 
implements ClusterChangeHan
 () -> (long) 
finalQueryQuotaEntity.getMaxQpsTracker().getMaxCountPerBucket());
   }
 
+  /**
+   * Add the query quota capacity utilization rate table gauge to the metric 
system if the qps quota is specified.
+   */
+  private void addQueryQuotaCapacityUtilizationRateTableGaugeIfNeeded(String 
tableNameWithType,
+  QueryQuotaEntity queryQuotaEntity) {
+if (queryQuotaEntity.getRateLimiter() != null) {
+  final QueryQuotaEntity finalQueryQuotaEntity = queryQuotaEntity;
+  _brokerMetrics.setOrUpdateTableGauge(tableNameWithType, 
BrokerGauge.QUERY_QUOTA_CAPACITY_UTILIZATION_RATE, () -> {
+double perBrokerRate = 
finalQueryQuotaEntity.getRateLimiter().getRate();
+int actualHitCountWithinTimeRange = 
finalQueryQuotaEntity.getMaxQpsTracker().getHitCount();
+long hitCountAllowedWithinTimeRage =
+(long) (perBrokerRate * 
finalQueryQuotaEntity.getMaxQpsTracker().getDefaultTimeRangeMs() / 1000L);
+// Since the MaxQpsTracker specifies 1-min window as valid time range, 
we can get the query quota capacity
+// utilization by using the actual hit count within 1 min divided by 
the expected hit count within 1 min.
+long percentageOfCapacityUtilization = actualHitCountWithinTimeRange * 
100L / hitCountAllowedWithinTimeRage;
+LOGGER.debug("The percentage of rate limit capacity utilization is 
{}", percentageOfCapacityUtilization);
+return percentageOfCapacityUtilization;
+  });
+}
+  }
+
   /**
* {@inheritDoc}
* Acquires a token from rate limiter based on the table name.
@@ -316,13 +339,6 @@ public class HelixExternalViewBasedQueryQuotaManager 
implements ClusterChangeHan
 
 // Emit the qps capacity utilization rate.
 int numHits = queryQuotaEntity.getQpsTracker().getHitCount();
-if (_brokerMetrics != null) {
-  int percentageOfCapacityUtilization = (int) (numHits * 100 / 
perBrokerRate);
-  LOGGER.debug("The percentage of rate limit capacity utilization is {}", 
percentageOfCapacityUtilization);
-  _brokerMetrics.setValueOfTableGauge(tableNameWithType, 
BrokerGauge.QUERY_QUOTA_CAPACITY_UTILIZATION_RATE,
-  percentageOfCapacityUtilization);
-}
-
 if (!rateLimiter.tryAcquire()) {
   LOGGER.info("Quota is exceeded for table: {}. Per-broker rate: {}. 
Current qps: {}", tableNameWithType,
   perBrokerRate, numHits);
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HitCounter.java 
b/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HitCounter.java
index eedc53903d..b656c02344 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HitCounter.java
+++ 
b/pinot-broker/sr

(pinot) branch emit-table-size-metrics updated (3183940a20 -> ffd1404b8e)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch emit-table-size-metrics
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 3183940a20 Emit table size related metrics only in lead controller
 add ffd1404b8e Emit table size related metrics only in lead controller

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (3183940a20)
\
 N -- N -- N   refs/heads/emit-table-size-metrics (ffd1404b8e)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/pinot/controller/api/TableSizeReaderTest.java | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch emit-table-size-metrics created (now 3183940a20)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch emit-table-size-metrics
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 3183940a20 Emit table size related metrics only in lead controller

This branch includes the following new commits:

 new 3183940a20 Emit table size related metrics only in lead controller

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Emit table size related metrics only in lead controller

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch emit-table-size-metrics
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 3183940a2078e6de58642f2a2b9cd463f3d19f57
Author: jlli_LinkedIn 
AuthorDate: Thu Mar 28 20:42:23 2024 -0700

Emit table size related metrics only in lead controller
---
 .../controller/api/resources/DebugResource.java|  7 +++-
 .../PinotSegmentUploadDownloadRestletResource.java |  2 +-
 .../pinot/controller/api/resources/TableSize.java  |  7 +++-
 .../api/upload/SegmentValidationUtils.java |  6 ++--
 .../controller/helix/SegmentStatusChecker.java |  2 +-
 .../pinot/controller/util/TableSizeReader.java | 41 --
 .../pinot/controller/api/TableSizeReaderTest.java  | 11 --
 7 files changed, 49 insertions(+), 27 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/DebugResource.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/DebugResource.java
index 88d81f679f..e8dbd0d8a1 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/DebugResource.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/DebugResource.java
@@ -65,6 +65,7 @@ import 
org.apache.pinot.common.restlet.resources.SegmentErrorInfo;
 import org.apache.pinot.common.restlet.resources.SegmentServerDebugInfo;
 import org.apache.pinot.common.utils.DatabaseUtils;
 import org.apache.pinot.controller.ControllerConf;
+import org.apache.pinot.controller.LeadControllerManager;
 import org.apache.pinot.controller.api.debug.TableDebugInfo;
 import 
org.apache.pinot.controller.api.exception.ControllerApplicationException;
 import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
@@ -123,6 +124,9 @@ public class DebugResource {
   @Inject
   ControllerConf _controllerConf;
 
+  @Inject
+  LeadControllerManager _leadControllerManager;
+
   @GET
   @Path("tables/{tableName}")
   @Authorize(targetType = TargetType.TABLE, paramName = "tableName", action = 
Actions.Table.GET_DEBUG_INFO)
@@ -238,7 +242,8 @@ public class DebugResource {
 
   private TableDebugInfo.TableSizeSummary getTableSize(String 
tableNameWithType) {
 TableSizeReader tableSizeReader =
-new TableSizeReader(_executor, _connectionManager, _controllerMetrics, 
_pinotHelixResourceManager);
+new TableSizeReader(_executor, _connectionManager, _controllerMetrics, 
_pinotHelixResourceManager,
+_leadControllerManager);
 TableSizeReader.TableSizeDetails tableSizeDetails;
 try {
   tableSizeDetails = tableSizeReader
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
index c92aef2cf3..04ff031674 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
@@ -363,7 +363,7 @@ public class PinotSegmentUploadDownloadRestletResource {
   }
   SegmentValidationUtils.checkStorageQuota(segmentName, 
untarredSegmentSizeInBytes, tableConfig,
   _pinotHelixResourceManager, _controllerConf, _controllerMetrics, 
_connectionManager, _executor,
-  _leadControllerManager.isLeaderForTable(tableNameWithType));
+  _leadControllerManager);
 
   // Encrypt segment
   String crypterNameInTableConfig = 
tableConfig.getValidationConfig().getCrypterClassName();
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/TableSize.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/TableSize.java
index dd8f7f78d3..22e95cc5f3 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/TableSize.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/TableSize.java
@@ -39,6 +39,7 @@ import javax.ws.rs.core.Response;
 import org.apache.http.conn.HttpClientConnectionManager;
 import org.apache.pinot.common.metrics.ControllerMetrics;
 import org.apache.pinot.controller.ControllerConf;
+import org.apache.pinot.controller.LeadControllerManager;
 import 
org.apache.pinot.controller.api.exception.ControllerApplicationException;
 import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
 import org.apache.pinot.controller.util.TableSizeReader;
@@ -70,6 +71,9 @@ public class TableSize {
   @Inject
   ControllerMetrics _controllerMetrics;
 
+  @Inject
+  LeadControllerManager _leadControllerManager;
+
   @GET
   @Path("/tables/{tableName}/size")
   @Authorize(targetType = TargetType.TABLE, paramName = "tableName

(pinot) branch full-auto-oss-abstraction updated (2355959756 -> 3dba068168)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-oss-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 2355959756 Extract methods for Pinot table ideal state
 new 3dba068168 Extract methods for Pinot table ideal state

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (2355959756)
\
 N -- N -- N   refs/heads/full-auto-oss-abstraction (3dba068168)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/pinot/controller/BaseControllerStarter.java|  5 ++---
 .../java/org/apache/pinot/controller/ControllerConf.java  |  2 +-
 .../pinot/controller/helix/core/util/HelixSetupUtils.java | 15 +++
 3 files changed, 18 insertions(+), 4 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Extract methods for Pinot table ideal state

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-oss-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 3dba068168ad0ae9184d9586a15f7ea4a29721f2
Author: jlli_LinkedIn 
AuthorDate: Fri Mar 22 16:00:43 2024 -0700

Extract methods for Pinot table ideal state
---
 .../MultiStageReplicaGroupSelector.java|  13 +-
 .../FullAutoInstancePartitionsUtils.java   |  52 +++
 .../common/assignment/InstancePartitionsUtils.java |  34 +++--
 .../assignment/InstancePartitionsUtilsHelper.java  |  58 
 .../InstancePartitionsUtilsHelperFactory.java  |  46 ++
 .../pinot/common/utils/config/TierConfigUtils.java |   7 +-
 .../pinot/controller/BaseControllerStarter.java|  10 ++
 .../apache/pinot/controller/ControllerConf.java|  11 ++
 .../PinotInstanceAssignmentRestletResource.java|  59 
 .../api/resources/PinotTenantRestletResource.java  |  12 +-
 .../helix/core/PinotHelixResourceManager.java  |  98 +---
 .../helix/core/PinotTableIdealStateHelper.java | 145 --
 .../instance/InstanceAssignmentDriver.java |   8 +-
 .../DefaultPinotTableIdealStateHelper.java | 151 +++
 .../FullAutoPinotTableIdealStateHelper.java| 165 +
 .../PinotTableIdealStateHelper.java|  87 +++
 .../PinotTableIdealStateHelperFactory.java |  50 +++
 .../realtime/MissingConsumingSegmentFinder.java|   4 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  97 +++-
 .../helix/core/rebalance/TableRebalancer.java  |  80 +-
 .../helix/core/util/HelixSetupUtils.java   |  15 ++
 ...anceAssignmentRestletResourceStatelessTest.java |   4 +-
 ...NonReplicaGroupTieredSegmentAssignmentTest.java |   8 +-
 ...NonReplicaGroupTieredSegmentAssignmentTest.java |   8 +-
 .../PinotLLCRealtimeSegmentManagerTest.java|   6 +-
 .../TableRebalancerClusterStatelessTest.java   |   4 +-
 .../helix/core/retention/RetentionManagerTest.java |  10 +-
 .../tools/admin/command/MoveReplicaGroup.java  |  20 +--
 28 files changed, 845 insertions(+), 417 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/MultiStageReplicaGroupSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/MultiStageReplicaGroupSelector.java
index 15fb525a8c..3f794c4017 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/MultiStageReplicaGroupSelector.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/MultiStageReplicaGroupSelector.java
@@ -34,7 +34,7 @@ import org.apache.helix.store.zk.ZkHelixPropertyStore;
 import org.apache.helix.zookeeper.datamodel.ZNRecord;
 import 
org.apache.pinot.broker.routing.adaptiveserverselector.AdaptiveServerSelector;
 import org.apache.pinot.common.assignment.InstancePartitions;
-import org.apache.pinot.common.assignment.InstancePartitionsUtils;
+import org.apache.pinot.common.assignment.InstancePartitionsUtilsHelperFactory;
 import org.apache.pinot.common.metrics.BrokerMetrics;
 import org.apache.pinot.spi.config.table.TableType;
 import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
@@ -145,12 +145,13 @@ public class MultiStageReplicaGroupSelector extends 
BaseInstanceSelector {
 Preconditions.checkNotNull(tableType);
 InstancePartitions instancePartitions;
 if (tableType.equals(TableType.OFFLINE)) {
-  instancePartitions = 
InstancePartitionsUtils.fetchInstancePartitions(_propertyStore,
-  
InstancePartitionsUtils.getInstancePartitionsName(_tableNameWithType, 
tableType.name()));
+  instancePartitions = 
InstancePartitionsUtilsHelperFactory.create().fetchInstancePartitions(_propertyStore,
+  InstancePartitionsUtilsHelperFactory.create()
+  .getInstancePartitionsName(_tableNameWithType, 
tableType.name()));
 } else {
-  instancePartitions = 
InstancePartitionsUtils.fetchInstancePartitions(_propertyStore,
-  InstancePartitionsUtils.getInstancePartitionsName(_tableNameWithType,
-  InstancePartitionsType.CONSUMING.name()));
+  instancePartitions = 
InstancePartitionsUtilsHelperFactory.create().fetchInstancePartitions(_propertyStore,
+  InstancePartitionsUtilsHelperFactory.create()
+  .getInstancePartitionsName(_tableNameWithType, 
InstancePartitionsType.CONSUMING.name()));
 }
 Preconditions.checkNotNull(instancePartitions);
 return instancePartitions;
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/assignment/FullAutoInstancePartitionsUtils.java
 
b/pinot-common/src/main/java/org/apache/pinot/common/assignment/FullAutoInstancePartitionsUtils.java
new file mode 100644
index 00..2821cc1c99
--- /dev/null
+++ 
b/pinot-common/src/main/java/org/apache/pinot

(pinot) branch full-auto-oss-abstraction updated (168a5c5a50 -> 2355959756)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-oss-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 168a5c5a50 Extract methods for Pinot table ideal state
 new 2355959756 Extract methods for Pinot table ideal state

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (168a5c5a50)
\
 N -- N -- N   refs/heads/full-auto-oss-abstraction (2355959756)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../MultiStageReplicaGroupSelector.java| 13 ++--
 .../FullAutoInstancePartitionsUtils.java   | 52 ++
 .../common/assignment/InstancePartitionsUtils.java | 34 +
 .../assignment/InstancePartitionsUtilsHelper.java  | 58 
 .../InstancePartitionsUtilsHelperFactory.java  | 33 -
 .../pinot/common/utils/config/TierConfigUtils.java |  7 +-
 .../pinot/controller/BaseControllerStarter.java|  7 ++
 .../PinotInstanceAssignmentRestletResource.java| 59 
 .../api/resources/PinotTenantRestletResource.java  | 12 ++--
 .../helix/core/PinotHelixResourceManager.java  | 52 +++---
 .../instance/InstanceAssignmentDriver.java |  8 +--
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  4 +-
 .../helix/core/rebalance/TableRebalancer.java  | 80 +++---
 ...anceAssignmentRestletResourceStatelessTest.java |  4 +-
 ...NonReplicaGroupTieredSegmentAssignmentTest.java |  8 +--
 ...NonReplicaGroupTieredSegmentAssignmentTest.java |  8 +--
 .../TableRebalancerClusterStatelessTest.java   |  4 +-
 17 files changed, 287 insertions(+), 156 deletions(-)
 create mode 100644 
pinot-common/src/main/java/org/apache/pinot/common/assignment/FullAutoInstancePartitionsUtils.java
 create mode 100644 
pinot-common/src/main/java/org/apache/pinot/common/assignment/InstancePartitionsUtilsHelper.java
 copy 
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/IndexEntry.java
 => 
pinot-common/src/main/java/org/apache/pinot/common/assignment/InstancePartitionsUtilsHelperFactory.java
 (59%)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Extract methods for Pinot table ideal state

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-oss-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 23559597562bdfddcaf95c762a09d548716ff670
Author: jlli_LinkedIn 
AuthorDate: Fri Mar 22 16:00:43 2024 -0700

Extract methods for Pinot table ideal state
---
 .../MultiStageReplicaGroupSelector.java|  13 +-
 .../FullAutoInstancePartitionsUtils.java   |  52 +++
 .../common/assignment/InstancePartitionsUtils.java |  34 +++--
 .../assignment/InstancePartitionsUtilsHelper.java  |  58 
 .../InstancePartitionsUtilsHelperFactory.java  |  46 ++
 .../pinot/common/utils/config/TierConfigUtils.java |   7 +-
 .../pinot/controller/BaseControllerStarter.java|  11 ++
 .../apache/pinot/controller/ControllerConf.java|  11 ++
 .../PinotInstanceAssignmentRestletResource.java|  59 
 .../api/resources/PinotTenantRestletResource.java  |  12 +-
 .../helix/core/PinotHelixResourceManager.java  |  98 +---
 .../helix/core/PinotTableIdealStateHelper.java | 145 --
 .../instance/InstanceAssignmentDriver.java |   8 +-
 .../DefaultPinotTableIdealStateHelper.java | 151 +++
 .../FullAutoPinotTableIdealStateHelper.java| 165 +
 .../PinotTableIdealStateHelper.java|  87 +++
 .../PinotTableIdealStateHelperFactory.java |  50 +++
 .../realtime/MissingConsumingSegmentFinder.java|   4 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  97 +++-
 .../helix/core/rebalance/TableRebalancer.java  |  80 +-
 ...anceAssignmentRestletResourceStatelessTest.java |   4 +-
 ...NonReplicaGroupTieredSegmentAssignmentTest.java |   8 +-
 ...NonReplicaGroupTieredSegmentAssignmentTest.java |   8 +-
 .../PinotLLCRealtimeSegmentManagerTest.java|   6 +-
 .../TableRebalancerClusterStatelessTest.java   |   4 +-
 .../helix/core/retention/RetentionManagerTest.java |  10 +-
 .../tools/admin/command/MoveReplicaGroup.java  |  20 +--
 27 files changed, 831 insertions(+), 417 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/MultiStageReplicaGroupSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/MultiStageReplicaGroupSelector.java
index 15fb525a8c..3f794c4017 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/MultiStageReplicaGroupSelector.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/MultiStageReplicaGroupSelector.java
@@ -34,7 +34,7 @@ import org.apache.helix.store.zk.ZkHelixPropertyStore;
 import org.apache.helix.zookeeper.datamodel.ZNRecord;
 import 
org.apache.pinot.broker.routing.adaptiveserverselector.AdaptiveServerSelector;
 import org.apache.pinot.common.assignment.InstancePartitions;
-import org.apache.pinot.common.assignment.InstancePartitionsUtils;
+import org.apache.pinot.common.assignment.InstancePartitionsUtilsHelperFactory;
 import org.apache.pinot.common.metrics.BrokerMetrics;
 import org.apache.pinot.spi.config.table.TableType;
 import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
@@ -145,12 +145,13 @@ public class MultiStageReplicaGroupSelector extends 
BaseInstanceSelector {
 Preconditions.checkNotNull(tableType);
 InstancePartitions instancePartitions;
 if (tableType.equals(TableType.OFFLINE)) {
-  instancePartitions = 
InstancePartitionsUtils.fetchInstancePartitions(_propertyStore,
-  
InstancePartitionsUtils.getInstancePartitionsName(_tableNameWithType, 
tableType.name()));
+  instancePartitions = 
InstancePartitionsUtilsHelperFactory.create().fetchInstancePartitions(_propertyStore,
+  InstancePartitionsUtilsHelperFactory.create()
+  .getInstancePartitionsName(_tableNameWithType, 
tableType.name()));
 } else {
-  instancePartitions = 
InstancePartitionsUtils.fetchInstancePartitions(_propertyStore,
-  InstancePartitionsUtils.getInstancePartitionsName(_tableNameWithType,
-  InstancePartitionsType.CONSUMING.name()));
+  instancePartitions = 
InstancePartitionsUtilsHelperFactory.create().fetchInstancePartitions(_propertyStore,
+  InstancePartitionsUtilsHelperFactory.create()
+  .getInstancePartitionsName(_tableNameWithType, 
InstancePartitionsType.CONSUMING.name()));
 }
 Preconditions.checkNotNull(instancePartitions);
 return instancePartitions;
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/assignment/FullAutoInstancePartitionsUtils.java
 
b/pinot-common/src/main/java/org/apache/pinot/common/assignment/FullAutoInstancePartitionsUtils.java
new file mode 100644
index 00..2821cc1c99
--- /dev/null
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/assignment/FullAutoInstancePartitionsUtils.java

(pinot) branch full-auto-oss-abstraction updated (f0e458d1ba -> 168a5c5a50)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-oss-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard f0e458d1ba Extract methods for Pinot table ideal state
 new 168a5c5a50 Extract methods for Pinot table ideal state

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (f0e458d1ba)
\
 N -- N -- N   refs/heads/full-auto-oss-abstraction (168a5c5a50)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/pinot/controller/ControllerConf.java|  11 +++
 .../helix/core/PinotHelixResourceManager.java  |  32 +-
 .../DefaultPinotTableIdealStateHelper.java |  85 +++-
 .../FullAutoPinotTableIdealStateHelper.java| 108 +
 .../PinotTableIdealStateHelper.java|  32 +-
 .../PinotTableIdealStateHelperFactory.java |  32 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  89 -
 .../PinotLLCRealtimeSegmentManagerTest.java|   6 +-
 .../tools/admin/command/MoveReplicaGroup.java  |  20 ++--
 9 files changed, 287 insertions(+), 128 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Extract methods for Pinot table ideal state

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-oss-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 168a5c5a50d9d5d2d8aaadf64bd415def5818793
Author: jlli_LinkedIn 
AuthorDate: Fri Mar 22 16:00:43 2024 -0700

Extract methods for Pinot table ideal state
---
 .../pinot/controller/BaseControllerStarter.java|   4 +
 .../apache/pinot/controller/ControllerConf.java|  11 ++
 .../helix/core/PinotHelixResourceManager.java  |  46 ++
 .../helix/core/PinotTableIdealStateHelper.java | 145 --
 .../DefaultPinotTableIdealStateHelper.java | 151 +++
 .../FullAutoPinotTableIdealStateHelper.java| 165 +
 .../PinotTableIdealStateHelper.java|  87 +++
 .../PinotTableIdealStateHelperFactory.java |  50 +++
 .../realtime/MissingConsumingSegmentFinder.java|   4 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  93 +++-
 .../PinotLLCRealtimeSegmentManagerTest.java|   6 +-
 .../helix/core/retention/RetentionManagerTest.java |  10 +-
 .../tools/admin/command/MoveReplicaGroup.java  |  20 +--
 13 files changed, 513 insertions(+), 279 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java
index 281c397401..c509738fca 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java
@@ -88,6 +88,7 @@ import 
org.apache.pinot.controller.helix.RealtimeConsumerMonitor;
 import org.apache.pinot.controller.helix.SegmentStatusChecker;
 import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
 import 
org.apache.pinot.controller.helix.core.cleanup.StaleInstancesCleanupTask;
+import 
org.apache.pinot.controller.helix.core.idealstatehelper.PinotTableIdealStateHelperFactory;
 import 
org.apache.pinot.controller.helix.core.minion.PinotHelixTaskResourceManager;
 import org.apache.pinot.controller.helix.core.minion.PinotTaskManager;
 import org.apache.pinot.controller.helix.core.minion.TaskMetricsEmitter;
@@ -245,6 +246,9 @@ public abstract class BaseControllerStarter implements 
ServiceStartable {
   _tenantRebalancer = new DefaultTenantRebalancer(_helixResourceManager, 
_tenantRebalanceExecutorService);
 }
 
+// Initialize the ideal state helper for Pinot tables.
+PinotTableIdealStateHelperFactory.init(_config);
+
 // Initialize the table config tuner registry.
 TableConfigTunerRegistry.init(_config.getTableConfigTunerPackages());
   }
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
index 4598b48eeb..8454aeb4ff 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/ControllerConf.java
@@ -275,6 +275,7 @@ public class ControllerConf extends PinotConfiguration {
   public static final String ACCESS_CONTROL_USERNAME = 
"access.control.init.username";
   public static final String ACCESS_CONTROL_PASSWORD = 
"access.control.init.password";
   public static final String LINEAGE_MANAGER_CLASS = 
"controller.lineage.manager.class";
+  public static final String PINOT_TABLE_IDEALSTATE_HELPER_CLASS = 
"controller.pinot.table.idealstate.class";
   // Amount of the time the segment can take from the beginning of upload to 
the end of upload. Used when parallel push
   // protection is enabled. If the upload does not finish within the timeout, 
next upload can override the previous one.
   private static final String SEGMENT_UPLOAD_TIMEOUT_IN_MILLIS = 
"controller.segment.upload.timeoutInMillis";
@@ -298,6 +299,8 @@ public class ControllerConf extends PinotConfiguration {
   private static final String DEFAULT_ACCESS_CONTROL_PASSWORD = "admin";
   private static final String DEFAULT_LINEAGE_MANAGER =
   "org.apache.pinot.controller.helix.core.lineage.DefaultLineageManager";
+  private static final String DEFAULT_PINOT_TABLE_IDEALSTATE_HELPER_CLASS =
+  "org.apache.pinot.controller.helix.core.idealstatehelper";
   private static final long DEFAULT_SEGMENT_UPLOAD_TIMEOUT_IN_MILLIS = 
600_000L; // 10 minutes
   private static final int DEFAULT_MIN_NUM_CHARS_IN_IS_TO_TURN_ON_COMPRESSION 
= -1;
   private static final int DEFAULT_REALTIME_SEGMENT_METADATA_COMMIT_NUMLOCKS = 
64;
@@ -872,6 +875,14 @@ public class ControllerConf extends PinotConfiguration {
 setProperty(LINEAGE_MANAGER_CLASS, lineageModifierClass);
   }
 
+  public String getPinotTableIdeal

(pinot) 01/01: Extract methods for Pinot table ideal state

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-oss-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit f0e458d1baec2deb7084a2f37afcee3ee586ceac
Author: jlli_LinkedIn 
AuthorDate: Fri Mar 22 16:00:43 2024 -0700

Extract methods for Pinot table ideal state
---
 .../pinot/controller/BaseControllerStarter.java|   4 +
 .../helix/core/PinotHelixResourceManager.java  |  14 +-
 .../helix/core/PinotTableIdealStateHelper.java | 145 -
 .../DefaultPinotTableIdealStateHelper.java |  76 +++
 .../FullAutoPinotTableIdealStateHelper.java|  57 
 .../PinotTableIdealStateHelper.java|  57 
 .../PinotTableIdealStateHelperFactory.java |  26 
 .../realtime/MissingConsumingSegmentFinder.java|   4 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |   4 +-
 .../helix/core/retention/RetentionManagerTest.java |  10 +-
 10 files changed, 236 insertions(+), 161 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java
index 281c397401..c509738fca 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/BaseControllerStarter.java
@@ -88,6 +88,7 @@ import 
org.apache.pinot.controller.helix.RealtimeConsumerMonitor;
 import org.apache.pinot.controller.helix.SegmentStatusChecker;
 import org.apache.pinot.controller.helix.core.PinotHelixResourceManager;
 import 
org.apache.pinot.controller.helix.core.cleanup.StaleInstancesCleanupTask;
+import 
org.apache.pinot.controller.helix.core.idealstatehelper.PinotTableIdealStateHelperFactory;
 import 
org.apache.pinot.controller.helix.core.minion.PinotHelixTaskResourceManager;
 import org.apache.pinot.controller.helix.core.minion.PinotTaskManager;
 import org.apache.pinot.controller.helix.core.minion.TaskMetricsEmitter;
@@ -245,6 +246,9 @@ public abstract class BaseControllerStarter implements 
ServiceStartable {
   _tenantRebalancer = new DefaultTenantRebalancer(_helixResourceManager, 
_tenantRebalanceExecutorService);
 }
 
+// Initialize the ideal state helper for Pinot tables.
+PinotTableIdealStateHelperFactory.init(_config);
+
 // Initialize the table config tuner registry.
 TableConfigTunerRegistry.init(_config.getTableConfigTunerPackages());
   }
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
index 57c75d7618..1fa664b062 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
@@ -142,6 +142,8 @@ import 
org.apache.pinot.controller.helix.core.assignment.instance.InstanceAssign
 import 
org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignment;
 import 
org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignmentFactory;
 import 
org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignmentUtils;
+import 
org.apache.pinot.controller.helix.core.idealstatehelper.PinotTableIdealStateHelper;
+import 
org.apache.pinot.controller.helix.core.idealstatehelper.PinotTableIdealStateHelperFactory;
 import org.apache.pinot.controller.helix.core.lineage.LineageManager;
 import org.apache.pinot.controller.helix.core.lineage.LineageManagerFactory;
 import 
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager;
@@ -232,11 +234,14 @@ public class PinotHelixResourceManager {
   private SegmentDeletionManager _segmentDeletionManager;
   private PinotLLCRealtimeSegmentManager _pinotLLCRealtimeSegmentManager;
   private TableCache _tableCache;
+
+  private final PinotTableIdealStateHelper _pinotTableIdealStateHelper;
   private final LineageManager _lineageManager;
 
   public PinotHelixResourceManager(String zkURL, String helixClusterName, 
@Nullable String dataDir,
   boolean isSingleTenantCluster, boolean enableBatchMessageMode, int 
deletedSegmentsRetentionInDays,
-  boolean enableTieredSegmentAssignment, LineageManager lineageManager) {
+  boolean enableTieredSegmentAssignment, PinotTableIdealStateHelper 
pinotTableIdealStateHelper,
+  LineageManager lineageManager) {
 _helixZkURL = HelixConfig.getAbsoluteZkPathForHelix(zkURL);
 _helixClusterName = helixClusterName;
 _dataDir = dataDir;
@@ -258,6 +263,7 @@ public class PinotHelixResourceManager {
 for (int i = 0; i < _tableUpdaterLocks.length; i++) {
   _tableUpdaterLocks[i] = new Object();
 }
+_pinotTableIdealStateHel

(pinot) branch full-auto-oss-abstraction created (now f0e458d1ba)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-oss-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at f0e458d1ba Extract methods for Pinot table ideal state

This branch includes the following new commits:

 new f0e458d1ba Extract methods for Pinot table ideal state

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Leverage ONLINE-OFFLINE state model for realtime tables

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-same-state-model
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 4ea45ce08f423a9d9428ac1c752ae45ede921e05
Author: jlli_LinkedIn 
AuthorDate: Wed Mar 6 12:57:45 2024 -0800

Leverage ONLINE-OFFLINE state model for realtime tables
---
 .../pinot/broker/routing/BrokerRoutingManager.java |  3 +-
 .../instanceselector/BaseInstanceSelector.java | 25 ++-
 .../SegmentPartitionMetadataManager.java   |  2 +-
 .../instanceselector/InstanceSelectorTest.java | 13 ++--
 .../helix/core/PinotHelixResourceManager.java  | 25 ---
 .../helix/core/PinotTableIdealStateHelper.java |  2 +-
 .../segment/RealtimeSegmentAssignment.java |  5 +-
 .../assignment/segment/SegmentAssignmentUtils.java | 22 --
 .../segment/StrictRealtimeSegmentAssignment.java   |  8 +--
 .../realtime/MissingConsumingSegmentFinder.java| 23 --
 .../realtime/PinotLLCRealtimeSegmentManager.java   | 82 --
 ...altimeNonReplicaGroupSegmentAssignmentTest.java |  4 +-
 ...NonReplicaGroupTieredSegmentAssignmentTest.java |  6 +-
 .../RealtimeReplicaGroupSegmentAssignmentTest.java |  4 +-
 .../StrictRealtimeSegmentAssignmentTest.java   |  2 +-
 .../PinotLLCRealtimeSegmentManagerTest.java| 23 +++---
 .../helix/core/rebalance/TableRebalancerTest.java  |  7 +-
 .../realtime/RealtimeSegmentDataManager.java   | 37 --
 .../manager/realtime/RealtimeTableDataManager.java |  8 +--
 .../realtime/RealtimeSegmentDataManagerTest.java   | 15 ++--
 .../ControllerPeriodicTasksIntegrationTest.java|  2 +-
 .../tests/LLCRealtimeClusterIntegrationTest.java   |  4 +-
 ...PartialUpsertTableRebalanceIntegrationTest.java |  2 +-
 .../UpsertTableSegmentPreloadIntegrationTest.java  |  2 +-
 .../UpsertTableSegmentUploadIntegrationTest.java   |  2 +-
 .../local/utils/tablestate/TableStateUtils.java| 25 +--
 .../pinot/server/api/resources/TablesResource.java | 62 ++--
 .../server/starter/helix/BaseServerStarter.java| 17 -
 ...flineSegmentOnlineOfflineStateModelFactory.java | 11 +--
 .../airlineStats_realtime_table_config.json|  3 +
 30 files changed, 273 insertions(+), 173 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
index cc3a5354ef..01e1296a26 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
@@ -543,8 +543,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
 Set onlineSegments = new 
HashSet<>(HashUtil.getHashMapCapacity(segmentAssignment.size()));
 for (Map.Entry> entry : 
segmentAssignment.entrySet()) {
   Map instanceStateMap = entry.getValue();
-  if (instanceStateMap.containsValue(SegmentStateModel.ONLINE) || 
instanceStateMap.containsValue(
-  SegmentStateModel.CONSUMING)) {
+  if (instanceStateMap.containsValue(SegmentStateModel.ONLINE)) {
 onlineSegments.add(entry.getKey());
   }
 }
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
index de0a5f9600..492c920669 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
@@ -27,7 +27,6 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.SortedMap;
-import java.util.SortedSet;
 import java.util.TreeMap;
 import java.util.TreeSet;
 import javax.annotation.Nullable;
@@ -122,7 +121,7 @@ abstract class BaseInstanceSelector implements 
InstanceSelector {
* Returns whether the instance state is online for routing purpose 
(ONLINE/CONSUMING).
*/
   static boolean isOnlineForRouting(@Nullable String state) {
-return SegmentStateModel.ONLINE.equals(state) || 
SegmentStateModel.CONSUMING.equals(state);
+return SegmentStateModel.ONLINE.equals(state);
   }
 
   /**
@@ -200,7 +199,7 @@ abstract class BaseInstanceSelector implements 
InstanceSelector {
 for (Map.Entry entry : 
idealStateInstanceStateMap.entrySet()) {
   String instance = entry.getKey();
   // NOTE: DO NOT check if EV matches IS because it is a valid state when 
EV is CONSUMING while IS is ONLINE
-  if (isOnlineForRouting(externalViewInstanceStateMap.get(instance))) {
+  if (isOnlineForRouting(entry.getValue()) && 
isOnlineForRouting(externalViewInstanceStateMap.get(instance))) {
 onlineInstances.add(instance);
   }

(pinot) branch full-auto-same-state-model updated (6c7302caf9 -> 4ea45ce08f)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-same-state-model
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 6c7302caf9 Leverage ONLINE-OFFLINE state model for realtime tables
 new 4ea45ce08f Leverage ONLINE-OFFLINE state model for realtime tables

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (6c7302caf9)
\
 N -- N -- N   refs/heads/full-auto-same-state-model (4ea45ce08f)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../instanceselector/BaseInstanceSelector.java | 23 ---
 .../realtime/RealtimeSegmentDataManager.java   | 34 ++
 .../manager/realtime/RealtimeTableDataManager.java |  6 ++--
 .../realtime/RealtimeSegmentDataManagerTest.java   | 15 ++
 .../airlineStats_realtime_table_config.json|  3 ++
 5 files changed, 51 insertions(+), 30 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Leverage ONLINE-OFFLINE state model for realtime tables

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-same-state-model
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 6c7302caf904837f414e76f4ad77a78fdbd3e71c
Author: jlli_LinkedIn 
AuthorDate: Wed Mar 6 12:57:45 2024 -0800

Leverage ONLINE-OFFLINE state model for realtime tables
---
 .../pinot/broker/routing/BrokerRoutingManager.java |  3 +-
 .../instanceselector/BaseInstanceSelector.java |  2 +-
 .../SegmentPartitionMetadataManager.java   |  2 +-
 .../instanceselector/InstanceSelectorTest.java | 13 ++--
 .../helix/core/PinotHelixResourceManager.java  | 25 ---
 .../helix/core/PinotTableIdealStateHelper.java |  2 +-
 .../segment/RealtimeSegmentAssignment.java |  5 +-
 .../assignment/segment/SegmentAssignmentUtils.java | 22 --
 .../segment/StrictRealtimeSegmentAssignment.java   |  8 +--
 .../realtime/MissingConsumingSegmentFinder.java| 23 --
 .../realtime/PinotLLCRealtimeSegmentManager.java   | 82 --
 ...altimeNonReplicaGroupSegmentAssignmentTest.java |  4 +-
 ...NonReplicaGroupTieredSegmentAssignmentTest.java |  6 +-
 .../RealtimeReplicaGroupSegmentAssignmentTest.java |  4 +-
 .../StrictRealtimeSegmentAssignmentTest.java   |  2 +-
 .../PinotLLCRealtimeSegmentManagerTest.java| 23 +++---
 .../helix/core/rebalance/TableRebalancerTest.java  |  7 +-
 .../realtime/RealtimeSegmentDataManager.java   |  7 +-
 .../manager/realtime/RealtimeTableDataManager.java |  2 +-
 .../ControllerPeriodicTasksIntegrationTest.java|  2 +-
 .../tests/LLCRealtimeClusterIntegrationTest.java   |  4 +-
 ...PartialUpsertTableRebalanceIntegrationTest.java |  2 +-
 .../UpsertTableSegmentPreloadIntegrationTest.java  |  2 +-
 .../UpsertTableSegmentUploadIntegrationTest.java   |  2 +-
 .../local/utils/tablestate/TableStateUtils.java| 25 +--
 .../pinot/server/api/resources/TablesResource.java | 62 ++--
 .../server/starter/helix/BaseServerStarter.java| 17 -
 ...flineSegmentOnlineOfflineStateModelFactory.java | 11 +--
 28 files changed, 224 insertions(+), 145 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
index cc3a5354ef..01e1296a26 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
@@ -543,8 +543,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
 Set onlineSegments = new 
HashSet<>(HashUtil.getHashMapCapacity(segmentAssignment.size()));
 for (Map.Entry> entry : 
segmentAssignment.entrySet()) {
   Map instanceStateMap = entry.getValue();
-  if (instanceStateMap.containsValue(SegmentStateModel.ONLINE) || 
instanceStateMap.containsValue(
-  SegmentStateModel.CONSUMING)) {
+  if (instanceStateMap.containsValue(SegmentStateModel.ONLINE)) {
 onlineSegments.add(entry.getKey());
   }
 }
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
index de0a5f9600..91ee512f0c 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
@@ -122,7 +122,7 @@ abstract class BaseInstanceSelector implements 
InstanceSelector {
* Returns whether the instance state is online for routing purpose 
(ONLINE/CONSUMING).
*/
   static boolean isOnlineForRouting(@Nullable String state) {
-return SegmentStateModel.ONLINE.equals(state) || 
SegmentStateModel.CONSUMING.equals(state);
+return SegmentStateModel.ONLINE.equals(state);
   }
 
   /**
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java
index 3623955591..b3ba0ac28a 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java
@@ -127,7 +127,7 @@ public class SegmentPartitionMetadataManager implements 
SegmentZkMetadataFetchLi
 List onlineServers = new ArrayList<>(instanceStateMap.size());
 for (Map.Entry entry : instanceStateMap.entrySet()) {
   String instanceState = entry.getValue();
-  if (instanceState.equals(SegmentStateModel.ONLINE) || 
instanceState.equals(SegmentStateModel.CONSUMING)) {
+  if (insta

(pinot) branch full-auto-same-state-model updated (7408de4fd0 -> 6c7302caf9)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-same-state-model
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 7408de4fd0 Leverage ONLINE-OFFLINE state model for realtime tables
 new 6c7302caf9 Leverage ONLINE-OFFLINE state model for realtime tables

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (7408de4fd0)
\
 N -- N -- N   refs/heads/full-auto-same-state-model (6c7302caf9)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../realtime/PinotLLCRealtimeSegmentManager.java   | 37 --
 .../realtime/RealtimeSegmentDataManager.java   |  7 +++-
 2 files changed, 26 insertions(+), 18 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch full-auto-same-state-model created (now 7408de4fd0)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-same-state-model
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 7408de4fd0 Leverage ONLINE-OFFLINE state model for realtime tables

This branch includes the following new commits:

 new 7408de4fd0 Leverage ONLINE-OFFLINE state model for realtime tables

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Leverage ONLINE-OFFLINE state model for realtime tables

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-same-state-model
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 7408de4fd0069bf5caea938b4936d9f3a1f6a556
Author: jlli_LinkedIn 
AuthorDate: Wed Mar 6 12:57:45 2024 -0800

Leverage ONLINE-OFFLINE state model for realtime tables
---
 .../pinot/broker/routing/BrokerRoutingManager.java |  3 +-
 .../instanceselector/BaseInstanceSelector.java |  2 +-
 .../SegmentPartitionMetadataManager.java   |  2 +-
 .../instanceselector/InstanceSelectorTest.java | 13 +++--
 .../helix/core/PinotHelixResourceManager.java  | 25 +
 .../helix/core/PinotTableIdealStateHelper.java |  2 +-
 .../segment/RealtimeSegmentAssignment.java |  5 +-
 .../assignment/segment/SegmentAssignmentUtils.java | 22 ++--
 .../segment/StrictRealtimeSegmentAssignment.java   |  8 +--
 .../realtime/MissingConsumingSegmentFinder.java| 23 ++--
 .../realtime/PinotLLCRealtimeSegmentManager.java   | 57 ++--
 ...altimeNonReplicaGroupSegmentAssignmentTest.java |  4 +-
 ...NonReplicaGroupTieredSegmentAssignmentTest.java |  6 +--
 .../RealtimeReplicaGroupSegmentAssignmentTest.java |  4 +-
 .../StrictRealtimeSegmentAssignmentTest.java   |  2 +-
 .../PinotLLCRealtimeSegmentManagerTest.java| 23 
 .../helix/core/rebalance/TableRebalancerTest.java  |  7 ++-
 .../manager/realtime/RealtimeTableDataManager.java |  2 +-
 .../ControllerPeriodicTasksIntegrationTest.java|  2 +-
 .../tests/LLCRealtimeClusterIntegrationTest.java   |  4 +-
 ...PartialUpsertTableRebalanceIntegrationTest.java |  2 +-
 .../UpsertTableSegmentPreloadIntegrationTest.java  |  2 +-
 .../UpsertTableSegmentUploadIntegrationTest.java   |  2 +-
 .../local/utils/tablestate/TableStateUtils.java| 25 ++---
 .../pinot/server/api/resources/TablesResource.java | 62 ++
 .../server/starter/helix/BaseServerStarter.java| 17 --
 ...flineSegmentOnlineOfflineStateModelFactory.java | 11 ++--
 27 files changed, 204 insertions(+), 133 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
index cc3a5354ef..01e1296a26 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
@@ -543,8 +543,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
 Set onlineSegments = new 
HashSet<>(HashUtil.getHashMapCapacity(segmentAssignment.size()));
 for (Map.Entry> entry : 
segmentAssignment.entrySet()) {
   Map instanceStateMap = entry.getValue();
-  if (instanceStateMap.containsValue(SegmentStateModel.ONLINE) || 
instanceStateMap.containsValue(
-  SegmentStateModel.CONSUMING)) {
+  if (instanceStateMap.containsValue(SegmentStateModel.ONLINE)) {
 onlineSegments.add(entry.getKey());
   }
 }
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
index de0a5f9600..91ee512f0c 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
@@ -122,7 +122,7 @@ abstract class BaseInstanceSelector implements 
InstanceSelector {
* Returns whether the instance state is online for routing purpose 
(ONLINE/CONSUMING).
*/
   static boolean isOnlineForRouting(@Nullable String state) {
-return SegmentStateModel.ONLINE.equals(state) || 
SegmentStateModel.CONSUMING.equals(state);
+return SegmentStateModel.ONLINE.equals(state);
   }
 
   /**
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java
index 3623955591..b3ba0ac28a 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpartition/SegmentPartitionMetadataManager.java
@@ -127,7 +127,7 @@ public class SegmentPartitionMetadataManager implements 
SegmentZkMetadataFetchLi
 List onlineServers = new ArrayList<>(instanceStateMap.size());
 for (Map.Entry entry : instanceStateMap.entrySet()) {
   String instanceState = entry.getValue();
-  if (instanceState.equals(SegmentStateModel.ONLINE) || 
instanceState.equals(SegmentStateModel.CONSUMING)) {
+  if (instanceState.equals(SegmentSt

(pinot) 01/01: Initial POC code for hybrid table

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 025f388f44a66d44bdcbd7d9ccc707bc1ebd5fde
Author: jlli_LinkedIn 
AuthorDate: Mon Feb 12 23:36:47 2024 -0800

Initial POC code for hybrid table
---
 .../instanceselector/BaseInstanceSelector.java |  23 +-
 .../StrictReplicaGroupInstanceSelector.java|   4 +
 .../controller/helix/SegmentStatusChecker.java |   6 +-
 ...imeSegmentOnlineOfflineStateModelGenerator.java |  72 +
 .../helix/core/PinotHelixResourceManager.java  |  30 +-
 ...uilder.java => PinotTableIdealStateHelper.java} |  23 +-
 .../realtime/MissingConsumingSegmentFinder.java|  11 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  55 +++-
 .../helix/core/util/HelixSetupUtils.java   |  21 +-
 .../helix/core/retention/RetentionManagerTest.java |   6 +-
 .../realtime/RealtimeSegmentDataManager.java   |   7 +
 .../tests/LLCRealtimeClusterIntegrationTest.java   |   8 +-
 .../server/starter/helix/BaseServerStarter.java|  13 +-
 ...ltimeSegmentOnlineOfflineStateModelFactory.java | 325 +
 .../airlineStats_realtime_table_config.json|   4 +-
 15 files changed, 544 insertions(+), 64 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
index b2961eef94..de0a5f9600 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
@@ -27,6 +27,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.SortedMap;
+import java.util.SortedSet;
 import java.util.TreeMap;
 import java.util.TreeSet;
 import javax.annotation.Nullable;
@@ -199,7 +200,7 @@ abstract class BaseInstanceSelector implements 
InstanceSelector {
 for (Map.Entry entry : 
idealStateInstanceStateMap.entrySet()) {
   String instance = entry.getKey();
   // NOTE: DO NOT check if EV matches IS because it is a valid state when 
EV is CONSUMING while IS is ONLINE
-  if (isOnlineForRouting(entry.getValue()) && 
isOnlineForRouting(externalViewInstanceStateMap.get(instance))) {
+  if (isOnlineForRouting(externalViewInstanceStateMap.get(instance))) {
 onlineInstances.add(instance);
   }
 }
@@ -217,6 +218,14 @@ abstract class BaseInstanceSelector implements 
InstanceSelector {
 }
   }
 
+  static SortedSet convertToSortedSet(Set set) {
+if (set instanceof SortedSet) {
+  return (SortedSet) set;
+} else {
+  return new TreeSet<>(set);
+}
+  }
+
   /**
* Updates the segment maps based on the given ideal state, external view, 
online segments (segments with
* ONLINE/CONSUMING instances in the ideal state and pre-selected by the 
{@link SegmentPreSelector}) and new segments.
@@ -230,19 +239,21 @@ abstract class BaseInstanceSelector implements 
InstanceSelector {
 _newSegmentStateMap = new 
HashMap<>(HashUtil.getHashMapCapacity(newSegmentCreationTimeMap.size()));
 
 Map> idealStateAssignment = 
idealState.getRecord().getMapFields();
+Set idealStateSegmentSet = idealState.getPartitionSet();
 Map> externalViewAssignment = 
externalView.getRecord().getMapFields();
 for (String segment : onlineSegments) {
   Map idealStateInstanceStateMap = 
idealStateAssignment.get(segment);
+
   Long newSegmentCreationTimeMs = newSegmentCreationTimeMap.get(segment);
   Map externalViewInstanceStateMap = 
externalViewAssignment.get(segment);
   if (externalViewInstanceStateMap == null) {
 if (newSegmentCreationTimeMs != null) {
   // New segment
-  List candidates = new 
ArrayList<>(idealStateInstanceStateMap.size());
-  for (Map.Entry entry : 
convertToSortedMap(idealStateInstanceStateMap).entrySet()) {
-if (isOnlineForRouting(entry.getValue())) {
-  candidates.add(new SegmentInstanceCandidate(entry.getKey(), 
false));
-}
+  List candidates = new 
ArrayList<>(Integer.parseInt(idealState.getReplicas()));
+  for (String segmentName : convertToSortedSet(idealStateSegmentSet)) {
+//if (isOnlineForRouting(entry.getValue())) {
+candidates.add(new SegmentInstanceCandidate(segmentName, false));
+//}
   }
   _newSegmentStateMap.put(segment, new 
NewSegmentState(newSegmentCreationTimeMs, candidates));
 } else {
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/StrictReplicaGroupInstanceSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/Stri

(pinot) branch full-auto-abstraction created (now 025f388f44)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-abstraction
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 025f388f44 Initial POC code for hybrid table

This branch includes the following new commits:

 new 025f388f44 Initial POC code for hybrid table

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Initial POC code for hybrid table

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 6781a10bf41f168aa80a48a823645446f228fcd7
Author: jlli_LinkedIn 
AuthorDate: Mon Feb 12 23:36:47 2024 -0800

Initial POC code for hybrid table
---
 .../pinot/broker/routing/BrokerRoutingManager.java |  11 +-
 .../instanceselector/BaseInstanceSelector.java |  59 ++--
 .../StrictReplicaGroupInstanceSelector.java|  34 ++-
 .../controller/helix/SegmentStatusChecker.java |   6 +-
 ...imeSegmentOnlineOfflineStateModelGenerator.java |  72 +
 .../helix/core/PinotHelixResourceManager.java  |  30 +-
 .../helix/core/PinotTableIdealStateBuilder.java|  17 +-
 .../realtime/MissingConsumingSegmentFinder.java|   7 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  51 +++-
 .../helix/core/util/HelixSetupUtils.java   |  21 +-
 .../realtime/RealtimeSegmentDataManager.java   |   7 +
 .../tests/LLCRealtimeClusterIntegrationTest.java   |   8 +-
 .../server/starter/helix/BaseServerStarter.java|  13 +-
 ...ltimeSegmentOnlineOfflineStateModelFactory.java | 325 +
 .../airlineStats_realtime_table_config.json|   4 +-
 15 files changed, 573 insertions(+), 92 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
index cc3a5354ef..b6be06653b 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
@@ -431,7 +431,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
   externalViewVersion = externalView.getRecord().getVersion();
 }
 
-Set onlineSegments = getOnlineSegments(idealState);
+Set onlineSegments = getOnlineSegments(idealState, externalView);
 
 SegmentPreSelector segmentPreSelector =
 SegmentPreSelectorFactory.getSegmentPreSelector(tableConfig, 
_propertyStore);
@@ -480,7 +480,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
 if (offlineTableExternalView == null) {
   offlineTableExternalView = new ExternalView(offlineTableName);
 }
-Set offlineTableOnlineSegments = 
getOnlineSegments(offlineTableIdealState);
+Set offlineTableOnlineSegments = 
getOnlineSegments(offlineTableIdealState, offlineTableExternalView);
 SegmentPreSelector offlineTableSegmentPreSelector =
 
SegmentPreSelectorFactory.getSegmentPreSelector(offlineTableConfig, 
_propertyStore);
 Set offlineTablePreSelectedOnlineSegments =
@@ -538,8 +538,11 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
   /**
* Returns the online segments (with ONLINE/CONSUMING instances) in the 
given ideal state.
*/
-  private static Set getOnlineSegments(IdealState idealState) {
+  private static Set getOnlineSegments(IdealState idealState, 
ExternalView externalView) {
 Map> segmentAssignment = 
idealState.getRecord().getMapFields();
+if (segmentAssignment.isEmpty()) {
+  segmentAssignment = externalView.getRecord().getMapFields();
+}
 Set onlineSegments = new 
HashSet<>(HashUtil.getHashMapCapacity(segmentAssignment.size()));
 for (Map.Entry> entry : 
segmentAssignment.entrySet()) {
   Map instanceStateMap = entry.getValue();
@@ -777,7 +780,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
 // inconsistency between components, which is fine because the 
inconsistency only exists for the newly changed
 // segments and only lasts for a very short time.
 void onAssignmentChange(IdealState idealState, ExternalView externalView) {
-  Set onlineSegments = getOnlineSegments(idealState);
+  Set onlineSegments = getOnlineSegments(idealState, externalView);
   Set preSelectedOnlineSegments = 
_segmentPreSelector.preSelect(onlineSegments);
   _segmentZkMetadataFetcher.onAssignmentChange(idealState, externalView, 
preSelectedOnlineSegments);
   _segmentSelector.onAssignmentChange(idealState, externalView, 
preSelectedOnlineSegments);
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
index b2961eef94..aebafa7741 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
@@ -27,6 +27,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.SortedMap;
+import java.util.Sorted

(pinot) branch full-auto-poc updated (37dc958fbc -> 6781a10bf4)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 37dc958fbc Initial POC code for hybrid table
 new 6781a10bf4 Initial POC code for hybrid table

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (37dc958fbc)
\
 N -- N -- N   refs/heads/full-auto-poc (6781a10bf4)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../helix/core/realtime/PinotLLCRealtimeSegmentManager.java   | 8 
 .../apache/pinot/controller/helix/core/util/HelixSetupUtils.java  | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch full-auto-poc updated (3c1ecd9495 -> 37dc958fbc)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 3c1ecd9495 Initial POC code for hybrid table
 new 37dc958fbc Initial POC code for hybrid table

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (3c1ecd9495)
\
 N -- N -- N   refs/heads/full-auto-poc (37dc958fbc)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/realtime/PinotLLCRealtimeSegmentManager.java| 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Initial POC code for hybrid table

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 37dc958fbc8cc33ab500167e43e78d764f239297
Author: jlli_LinkedIn 
AuthorDate: Mon Feb 12 23:36:47 2024 -0800

Initial POC code for hybrid table
---
 .../pinot/broker/routing/BrokerRoutingManager.java |  11 +-
 .../instanceselector/BaseInstanceSelector.java |  59 ++--
 .../StrictReplicaGroupInstanceSelector.java|  34 ++-
 .../controller/helix/SegmentStatusChecker.java |   6 +-
 ...imeSegmentOnlineOfflineStateModelGenerator.java |  72 +
 .../helix/core/PinotHelixResourceManager.java  |  30 +-
 .../helix/core/PinotTableIdealStateBuilder.java|  17 +-
 .../realtime/MissingConsumingSegmentFinder.java|   7 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  51 +++-
 .../helix/core/util/HelixSetupUtils.java   |  23 +-
 .../realtime/RealtimeSegmentDataManager.java   |   7 +
 .../tests/LLCRealtimeClusterIntegrationTest.java   |   8 +-
 .../server/starter/helix/BaseServerStarter.java|  13 +-
 ...ltimeSegmentOnlineOfflineStateModelFactory.java | 325 +
 .../airlineStats_realtime_table_config.json|   4 +-
 15 files changed, 574 insertions(+), 93 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
index cc3a5354ef..b6be06653b 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
@@ -431,7 +431,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
   externalViewVersion = externalView.getRecord().getVersion();
 }
 
-Set onlineSegments = getOnlineSegments(idealState);
+Set onlineSegments = getOnlineSegments(idealState, externalView);
 
 SegmentPreSelector segmentPreSelector =
 SegmentPreSelectorFactory.getSegmentPreSelector(tableConfig, 
_propertyStore);
@@ -480,7 +480,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
 if (offlineTableExternalView == null) {
   offlineTableExternalView = new ExternalView(offlineTableName);
 }
-Set offlineTableOnlineSegments = 
getOnlineSegments(offlineTableIdealState);
+Set offlineTableOnlineSegments = 
getOnlineSegments(offlineTableIdealState, offlineTableExternalView);
 SegmentPreSelector offlineTableSegmentPreSelector =
 
SegmentPreSelectorFactory.getSegmentPreSelector(offlineTableConfig, 
_propertyStore);
 Set offlineTablePreSelectedOnlineSegments =
@@ -538,8 +538,11 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
   /**
* Returns the online segments (with ONLINE/CONSUMING instances) in the 
given ideal state.
*/
-  private static Set getOnlineSegments(IdealState idealState) {
+  private static Set getOnlineSegments(IdealState idealState, 
ExternalView externalView) {
 Map> segmentAssignment = 
idealState.getRecord().getMapFields();
+if (segmentAssignment.isEmpty()) {
+  segmentAssignment = externalView.getRecord().getMapFields();
+}
 Set onlineSegments = new 
HashSet<>(HashUtil.getHashMapCapacity(segmentAssignment.size()));
 for (Map.Entry> entry : 
segmentAssignment.entrySet()) {
   Map instanceStateMap = entry.getValue();
@@ -777,7 +780,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
 // inconsistency between components, which is fine because the 
inconsistency only exists for the newly changed
 // segments and only lasts for a very short time.
 void onAssignmentChange(IdealState idealState, ExternalView externalView) {
-  Set onlineSegments = getOnlineSegments(idealState);
+  Set onlineSegments = getOnlineSegments(idealState, externalView);
   Set preSelectedOnlineSegments = 
_segmentPreSelector.preSelect(onlineSegments);
   _segmentZkMetadataFetcher.onAssignmentChange(idealState, externalView, 
preSelectedOnlineSegments);
   _segmentSelector.onAssignmentChange(idealState, externalView, 
preSelectedOnlineSegments);
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
index b2961eef94..aebafa7741 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
@@ -27,6 +27,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.SortedMap;
+import java.util.Sorted

(pinot) 01/01: Initial POC code for hybrid table

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 3c1ecd9495d9cb479b8ab026e7e5c7a77246a2d9
Author: jlli_LinkedIn 
AuthorDate: Mon Feb 12 23:36:47 2024 -0800

Initial POC code for hybrid table
---
 .../pinot/broker/routing/BrokerRoutingManager.java |  11 +-
 .../instanceselector/BaseInstanceSelector.java |  59 ++--
 .../StrictReplicaGroupInstanceSelector.java|  34 ++-
 .../controller/helix/SegmentStatusChecker.java |   6 +-
 ...imeSegmentOnlineOfflineStateModelGenerator.java |  72 +
 .../helix/core/PinotHelixResourceManager.java  |  30 +-
 .../helix/core/PinotTableIdealStateBuilder.java|  17 +-
 .../realtime/MissingConsumingSegmentFinder.java|   7 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  37 ++-
 .../helix/core/util/HelixSetupUtils.java   |  23 +-
 .../realtime/RealtimeSegmentDataManager.java   |   7 +
 .../tests/LLCRealtimeClusterIntegrationTest.java   |   8 +-
 .../server/starter/helix/BaseServerStarter.java|  13 +-
 ...ltimeSegmentOnlineOfflineStateModelFactory.java | 325 +
 .../airlineStats_realtime_table_config.json|   4 +-
 15 files changed, 560 insertions(+), 93 deletions(-)

diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
index cc3a5354ef..b6be06653b 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/BrokerRoutingManager.java
@@ -431,7 +431,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
   externalViewVersion = externalView.getRecord().getVersion();
 }
 
-Set onlineSegments = getOnlineSegments(idealState);
+Set onlineSegments = getOnlineSegments(idealState, externalView);
 
 SegmentPreSelector segmentPreSelector =
 SegmentPreSelectorFactory.getSegmentPreSelector(tableConfig, 
_propertyStore);
@@ -480,7 +480,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
 if (offlineTableExternalView == null) {
   offlineTableExternalView = new ExternalView(offlineTableName);
 }
-Set offlineTableOnlineSegments = 
getOnlineSegments(offlineTableIdealState);
+Set offlineTableOnlineSegments = 
getOnlineSegments(offlineTableIdealState, offlineTableExternalView);
 SegmentPreSelector offlineTableSegmentPreSelector =
 
SegmentPreSelectorFactory.getSegmentPreSelector(offlineTableConfig, 
_propertyStore);
 Set offlineTablePreSelectedOnlineSegments =
@@ -538,8 +538,11 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
   /**
* Returns the online segments (with ONLINE/CONSUMING instances) in the 
given ideal state.
*/
-  private static Set getOnlineSegments(IdealState idealState) {
+  private static Set getOnlineSegments(IdealState idealState, 
ExternalView externalView) {
 Map> segmentAssignment = 
idealState.getRecord().getMapFields();
+if (segmentAssignment.isEmpty()) {
+  segmentAssignment = externalView.getRecord().getMapFields();
+}
 Set onlineSegments = new 
HashSet<>(HashUtil.getHashMapCapacity(segmentAssignment.size()));
 for (Map.Entry> entry : 
segmentAssignment.entrySet()) {
   Map instanceStateMap = entry.getValue();
@@ -777,7 +780,7 @@ public class BrokerRoutingManager implements 
RoutingManager, ClusterChangeHandle
 // inconsistency between components, which is fine because the 
inconsistency only exists for the newly changed
 // segments and only lasts for a very short time.
 void onAssignmentChange(IdealState idealState, ExternalView externalView) {
-  Set onlineSegments = getOnlineSegments(idealState);
+  Set onlineSegments = getOnlineSegments(idealState, externalView);
   Set preSelectedOnlineSegments = 
_segmentPreSelector.preSelect(onlineSegments);
   _segmentZkMetadataFetcher.onAssignmentChange(idealState, externalView, 
preSelectedOnlineSegments);
   _segmentSelector.onAssignmentChange(idealState, externalView, 
preSelectedOnlineSegments);
diff --git 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
index b2961eef94..aebafa7741 100644
--- 
a/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
+++ 
b/pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java
@@ -27,6 +27,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.SortedMap;
+import java.util.Sorted

(pinot) branch full-auto-poc updated (e4acc6e70c -> 3c1ecd9495)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard e4acc6e70c Initial POC code for hybrid table
 new 3c1ecd9495 Initial POC code for hybrid table

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (e4acc6e70c)
\
 N -- N -- N   refs/heads/full-auto-poc (3c1ecd9495)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../pinot/broker/routing/BrokerRoutingManager.java | 11 ++--
 .../instanceselector/BaseInstanceSelector.java | 59 +-
 .../StrictReplicaGroupInstanceSelector.java| 34 +++--
 .../helix/core/util/HelixSetupUtils.java   |  2 +-
 4 files changed, 61 insertions(+), 45 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch full-auto-poc updated (35e96d1eb5 -> e4acc6e70c)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 35e96d1eb5 Initial POC code for hybrid table
 new e4acc6e70c Initial POC code for hybrid table

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (35e96d1eb5)
\
 N -- N -- N   refs/heads/full-auto-poc (e4acc6e70c)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../helix/core/realtime/PinotLLCRealtimeSegmentManager.java   | 7 +--
 .../integration/tests/LLCRealtimeClusterIntegrationTest.java  | 8 ++--
 .../helix/RealtimeSegmentOnlineOfflineStateModelFactory.java  | 2 +-
 3 files changed, 12 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Initial POC code for hybrid table

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit e4acc6e70c2be0125fbdd54dd3c2fe99db0d705d
Author: jlli_LinkedIn 
AuthorDate: Mon Feb 12 23:36:47 2024 -0800

Initial POC code for hybrid table
---
 .../controller/helix/SegmentStatusChecker.java |   6 +-
 ...imeSegmentOnlineOfflineStateModelGenerator.java |  72 +
 .../helix/core/PinotHelixResourceManager.java  |  30 +-
 .../helix/core/PinotTableIdealStateBuilder.java|  17 +-
 .../realtime/MissingConsumingSegmentFinder.java|   7 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  37 ++-
 .../helix/core/util/HelixSetupUtils.java   |  21 +-
 .../realtime/RealtimeSegmentDataManager.java   |   7 +
 .../tests/LLCRealtimeClusterIntegrationTest.java   |   8 +-
 .../server/starter/helix/BaseServerStarter.java|  13 +-
 ...ltimeSegmentOnlineOfflineStateModelFactory.java | 325 +
 .../airlineStats_realtime_table_config.json|   4 +-
 12 files changed, 499 insertions(+), 48 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java
index d0af31044f..2b9431b0f2 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java
@@ -256,8 +256,12 @@ public class SegmentStatusChecker extends 
ControllerPeriodicTask partitionMap = 
idealState.getInstanceStateMap(partitionName);
+  if (partitionMap == null) {
+continue;
+  }
   // Skip segments not online in ideal state
-  for (Map.Entry serverAndState : 
idealState.getInstanceStateMap(partitionName).entrySet()) {
+  for (Map.Entry serverAndState : partitionMap.entrySet()) 
{
 if (serverAndState == null) {
   break;
 }
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator.java
new file mode 100644
index 00..ec640131cb
--- /dev/null
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator.java
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core;
+
+import org.apache.helix.model.StateModelDefinition;
+
+
+/**
+ * Offline Segment state model generator describes the transitions for offline 
segment states.
+ *
+ * Online to Offline, Online to Dropped
+ * Offline to Online, Offline to Dropped
+ *
+ * This does not include the state transitions for realtime segments (which 
includes the CONSUMING state)
+ */
+public class PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator {
+  private PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator() {
+  }
+
+  public static final String PINOT_REALTIME_SEGMENT_ONLINE_OFFLINE_STATE_MODEL 
=
+  "RealtimeSegmentOnlineOfflineStateModel";
+
+  public static final String ONLINE_STATE = "ONLINE";
+  public static final String CONSUMING_STATE = "CONSUMING";
+  public static final String OFFLINE_STATE = "OFFLINE";
+  public static final String DROPPED_STATE = "DROPPED";
+
+  public static StateModelDefinition generatePinotStateModelDefinition() {
+StateModelDefinition.Builder builder =
+new 
StateModelDefinition.Builder(PINOT_REALTIME_SEGMENT_ONLINE_OFFLINE_STATE_MODEL);
+builder.initialState(OFFLINE_STATE);
+
+builder.addState(ONLINE_STATE);
+builder.addState(CONSUMING_STATE);
+builder.addState(OFFLINE_STATE);
+builder.addState(DROPPED_STATE);
+// Set the initial state when the node starts
+
+// Add transitions between the states.
+builder.addTransition(CONSUMING_STATE, ONLINE_STATE);
+builder.addTr

(pinot) branch full-auto-poc created (now 35e96d1eb5)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 35e96d1eb5 Initial POC code for hybrid table

This branch includes the following new commits:

 new 9864022a2f Initial POC code
 new 35e96d1eb5 Initial POC code for hybrid table

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/02: Initial POC code

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 9864022a2f0e9a7a8c5f2e3e6ab07ae89f658497
Author: Sonam Mandal 
AuthorDate: Wed Jan 17 13:31:58 2024 -0800

Initial POC code
---
 .../common/assignment/InstancePartitionsUtils.java |  11 +-
 .../pinot/common/utils/helix/HelixHelper.java  |  10 +-
 .../pinot/controller/LeadControllerManager.java|   1 +
 .../PinotInstanceAssignmentRestletResource.java|   4 +-
 .../api/resources/PinotTenantRestletResource.java  |   3 +-
 ...ineSegmentOnlineOfflineStateModelGenerator.java |  65 +
 .../helix/core/PinotHelixResourceManager.java  |  46 --
 ...lixSegmentOnlineOfflineStateModelGenerator.java |   2 +-
 .../helix/core/PinotTableIdealStateBuilder.java|  30 
 .../assignment/segment/SegmentAssignmentUtils.java |   8 +
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  50 ---
 .../helix/core/rebalance/TableRebalancer.java  |  46 +-
 .../helix/core/util/HelixSetupUtils.java   |  19 +++
 .../PinotLLCRealtimeSegmentManagerTest.java|   6 +-
 .../integration/tests/HelixZNodeSizeLimitTest.java |  19 ++-
 .../server/starter/helix/BaseServerStarter.java|   9 +-
 ...flineSegmentOnlineOfflineStateModelFactory.java | 162 +
 .../SegmentOnlineOfflineStateModelFactory.java |  38 +
 .../org/apache/pinot/tools/HybridQuickstart.java   |   5 +-
 .../tools/admin/command/MoveReplicaGroup.java  |  17 ++-
 .../tools/admin/command/QuickstartRunner.java  |   2 +-
 .../airlineStats_offline_table_config.json |   4 +-
 22 files changed, 501 insertions(+), 56 deletions(-)

diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/assignment/InstancePartitionsUtils.java
 
b/pinot-common/src/main/java/org/apache/pinot/common/assignment/InstancePartitionsUtils.java
index 759d387af4..f8bbd08934 100644
--- 
a/pinot-common/src/main/java/org/apache/pinot/common/assignment/InstancePartitionsUtils.java
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/assignment/InstancePartitionsUtils.java
@@ -23,7 +23,9 @@ import java.util.Collections;
 import java.util.List;
 import javax.annotation.Nullable;
 import org.apache.helix.AccessOption;
+import org.apache.helix.ConfigAccessor;
 import org.apache.helix.HelixManager;
+import org.apache.helix.model.ResourceConfig;
 import org.apache.helix.store.HelixPropertyStore;
 import org.apache.helix.store.zk.ZkHelixPropertyStore;
 import org.apache.helix.zookeeper.datamodel.ZNRecord;
@@ -165,12 +167,19 @@ public class InstancePartitionsUtils {
* Persists the instance partitions to Helix property store.
*/
   public static void persistInstancePartitions(HelixPropertyStore 
propertyStore,
-  InstancePartitions instancePartitions) {
+  ConfigAccessor configAccessor, String helixClusterName, 
InstancePartitions instancePartitions) {
 String path = ZKMetadataProvider
 
.constructPropertyStorePathForInstancePartitions(instancePartitions.getInstancePartitionsName());
 if (!propertyStore.set(path, instancePartitions.toZNRecord(), 
AccessOption.PERSISTENT)) {
   throw new ZkException("Failed to persist instance partitions: " + 
instancePartitions);
 }
+
+// Set the INSTANCE_PARTITIONS under the RESOURCE config (only modifying 
set path for now to ensure it works)
+// This is just a test to see how to access and update the CONFIG/RESOURCES
+String resourceName = "INSTANCE_PARTITION_" + 
instancePartitions.getInstancePartitionsName();
+ResourceConfig resourceConfig = new ResourceConfig(resourceName);
+
resourceConfig.setPreferenceLists(instancePartitions.getPartitionToInstancesMap());
+configAccessor.setResourceConfig(helixClusterName, resourceName, 
resourceConfig);
   }
 
   /**
diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
 
b/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
index 4160dd44ef..8f76ba801f 100644
--- 
a/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/utils/helix/HelixHelper.java
@@ -185,9 +185,12 @@ public class HelixHelper {
 String partitionName = it.next();
 int numChars = partitionName.length();
 Map stateMap = 
is.getInstanceStateMap(partitionName);
-for (Map.Entry entry : stateMap.entrySet()) {
-  numChars += entry.getKey().length();
-  numChars += entry.getValue().length();
+if (stateMap != null) {
+  // The stateMap might be NULL for FULL-AUTO segments, so always 
do this NULL check
+  for (Map.Entry entry : stateMap.entrySet()) {
+numChars += entry.getKey().length();
+numChars += ent

(pinot) 02/02: Initial POC code for hybrid table

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch full-auto-poc
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 35e96d1eb5a5662a62ab43dac4ad7ea9716f8b42
Author: jlli_LinkedIn 
AuthorDate: Mon Feb 12 23:36:47 2024 -0800

Initial POC code for hybrid table
---
 .../controller/helix/SegmentStatusChecker.java |   6 +-
 ...imeSegmentOnlineOfflineStateModelGenerator.java |  72 +
 .../helix/core/PinotHelixResourceManager.java  |  30 +-
 .../helix/core/PinotTableIdealStateBuilder.java|  17 +-
 .../realtime/MissingConsumingSegmentFinder.java|   7 +-
 .../realtime/PinotLLCRealtimeSegmentManager.java   |  30 +-
 .../helix/core/util/HelixSetupUtils.java   |  21 +-
 .../realtime/RealtimeSegmentDataManager.java   |   7 +
 .../server/starter/helix/BaseServerStarter.java|  13 +-
 ...ltimeSegmentOnlineOfflineStateModelFactory.java | 325 +
 .../airlineStats_realtime_table_config.json|   4 +-
 11 files changed, 488 insertions(+), 44 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java
index d0af31044f..2b9431b0f2 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/SegmentStatusChecker.java
@@ -256,8 +256,12 @@ public class SegmentStatusChecker extends 
ControllerPeriodicTask partitionMap = 
idealState.getInstanceStateMap(partitionName);
+  if (partitionMap == null) {
+continue;
+  }
   // Skip segments not online in ideal state
-  for (Map.Entry serverAndState : 
idealState.getInstanceStateMap(partitionName).entrySet()) {
+  for (Map.Entry serverAndState : partitionMap.entrySet()) 
{
 if (serverAndState == null) {
   break;
 }
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator.java
new file mode 100644
index 00..ec640131cb
--- /dev/null
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator.java
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core;
+
+import org.apache.helix.model.StateModelDefinition;
+
+
+/**
+ * Offline Segment state model generator describes the transitions for offline 
segment states.
+ *
+ * Online to Offline, Online to Dropped
+ * Offline to Online, Offline to Dropped
+ *
+ * This does not include the state transitions for realtime segments (which 
includes the CONSUMING state)
+ */
+public class PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator {
+  private PinotHelixRealtimeSegmentOnlineOfflineStateModelGenerator() {
+  }
+
+  public static final String PINOT_REALTIME_SEGMENT_ONLINE_OFFLINE_STATE_MODEL 
=
+  "RealtimeSegmentOnlineOfflineStateModel";
+
+  public static final String ONLINE_STATE = "ONLINE";
+  public static final String CONSUMING_STATE = "CONSUMING";
+  public static final String OFFLINE_STATE = "OFFLINE";
+  public static final String DROPPED_STATE = "DROPPED";
+
+  public static StateModelDefinition generatePinotStateModelDefinition() {
+StateModelDefinition.Builder builder =
+new 
StateModelDefinition.Builder(PINOT_REALTIME_SEGMENT_ONLINE_OFFLINE_STATE_MODEL);
+builder.initialState(OFFLINE_STATE);
+
+builder.addState(ONLINE_STATE);
+builder.addState(CONSUMING_STATE);
+builder.addState(OFFLINE_STATE);
+builder.addState(DROPPED_STATE);
+// Set the initial state when the node starts
+
+// Add transitions between the states.
+builder.addTransition(CONSUMING_STATE, ONLINE_STATE);
+builder.addTransition(OFFLINE_STATE, CONSUMING_STATE);
+//builder.

(pinot) branch master updated (71b72e13c4 -> 5c81c0a88a)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git


from 71b72e13c4 Fixing quickstart table baseballStats minion ingestion 
(#12371)
 add 5c81c0a88a Fix backward compatible issue in 
DistinctCountThetaSketchAggregationFunction (#12347)

No new revisions were added by this update.

Summary of changes:
 ...inctCountRawThetaSketchAggregationFunction.java |  3 ++-
 ...istinctCountThetaSketchAggregationFunction.java | 25 +-
 2 files changed, 12 insertions(+), 16 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch fix-DistinctCountThetaSketchAggregationFunction deleted (was db75e449ad)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-DistinctCountThetaSketchAggregationFunction
in repository https://gitbox.apache.org/repos/asf/pinot.git


 was db75e449ad Fix backward compatible issue in 
DistinctCountThetaSketchAggregationFunction

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch master updated: Release all segments of a table in releaseAndRemoveAllSegments method (#12297)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 87c089f8eb Release all segments of a table in 
releaseAndRemoveAllSegments method (#12297)
87c089f8eb is described below

commit 87c089f8eb4ea019231ac5a5600aaf12f591a20d
Author: Jialiang Li 
AuthorDate: Fri Feb 2 10:11:45 2024 -0800

Release all segments of a table in releaseAndRemoveAllSegments method 
(#12297)
---
 .../pinot/core/data/manager/BaseTableDataManager.java | 19 +--
 .../starter/helix/HelixInstanceDataManager.java   | 19 +--
 2 files changed, 34 insertions(+), 4 deletions(-)

diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
index 119bede805..191ea04a0d 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
@@ -33,6 +33,7 @@ import java.util.Map;
 import java.util.UUID;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
 import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
@@ -223,8 +224,22 @@ public abstract class BaseTableDataManager implements 
TableDataManager {
   segmentDataManagers = new ArrayList<>(_segmentDataManagerMap.values());
   _segmentDataManagerMap.clear();
 }
-for (SegmentDataManager segmentDataManager : segmentDataManagers) {
-  releaseSegment(segmentDataManager);
+if (!segmentDataManagers.isEmpty()) {
+  int numThreads = Math.min(Runtime.getRuntime().availableProcessors(), 
segmentDataManagers.size());
+  ExecutorService stopExecutorService = 
Executors.newFixedThreadPool(numThreads);
+  for (SegmentDataManager segmentDataManager : segmentDataManagers) {
+stopExecutorService.submit(() -> releaseSegment(segmentDataManager));
+  }
+  stopExecutorService.shutdown();
+  try {
+// Wait at most 10 minutes before exiting this method.
+if (!stopExecutorService.awaitTermination(10, TimeUnit.MINUTES)) {
+  stopExecutorService.shutdownNow();
+}
+  } catch (InterruptedException e) {
+stopExecutorService.shutdownNow();
+Thread.currentThread().interrupt();
+  }
 }
   }
 
diff --git 
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
 
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
index 8e147cc74a..f097c4103c 100644
--- 
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
+++ 
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
@@ -33,6 +33,7 @@ import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.concurrent.locks.Lock;
 import java.util.function.Supplier;
@@ -203,8 +204,22 @@ public class HelixInstanceDataManager implements 
InstanceDataManager {
 if (_segmentPreloadExecutor != null) {
   _segmentPreloadExecutor.shutdownNow();
 }
-for (TableDataManager tableDataManager : _tableDataManagerMap.values()) {
-  tableDataManager.shutDown();
+if (!_tableDataManagerMap.isEmpty()) {
+  int numThreads = Math.min(Runtime.getRuntime().availableProcessors(), 
_tableDataManagerMap.size());
+  ExecutorService stopExecutorService = 
Executors.newFixedThreadPool(numThreads);
+  for (TableDataManager tableDataManager : _tableDataManagerMap.values()) {
+stopExecutorService.submit(tableDataManager::shutDown);
+  }
+  stopExecutorService.shutdown();
+  try {
+// Wait at most 10 minutes before exiting this method.
+if (!stopExecutorService.awaitTermination(10, TimeUnit.MINUTES)) {
+  stopExecutorService.shutdownNow();
+}
+  } catch (InterruptedException e) {
+stopExecutorService.shutdownNow();
+Thread.currentThread().interrupt();
+  }
 }
 SegmentBuildTimeLeaseExtender.shutdownExecutor();
 LOGGER.info("Helix instance data manager shut down");


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments deleted (was a796baa1b5)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments
in repository https://gitbox.apache.org/repos/asf/pinot.git


 was a796baa1b5 Release all segments of a table in 
releaseAndRemoveAllSegments method

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments updated (4f05fd71ae -> a796baa1b5)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments
in repository https://gitbox.apache.org/repos/asf/pinot.git


omit 4f05fd71ae Release all segments of a table in 
releaseAndRemoveAllSegments method
 add a796baa1b5 Release all segments of a table in 
releaseAndRemoveAllSegments method

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (4f05fd71ae)
\
 N -- N -- N   
refs/heads/stop-all-segments-in-releaseAndRemoveAllSegments (a796baa1b5)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../starter/helix/HelixInstanceDataManager.java| 26 --
 1 file changed, 14 insertions(+), 12 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch fix-DistinctCountThetaSketchAggregationFunction updated (577e5e2c58 -> db75e449ad)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-DistinctCountThetaSketchAggregationFunction
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 577e5e2c58 Fix backward compatible issue in 
DistinctCountThetaSketchAggregationFunction
 add db75e449ad Fix backward compatible issue in 
DistinctCountThetaSketchAggregationFunction

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (577e5e2c58)
\
 N -- N -- N   
refs/heads/fix-DistinctCountThetaSketchAggregationFunction (db75e449ad)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 ...inctCountRawThetaSketchAggregationFunction.java |  3 ++-
 ...istinctCountThetaSketchAggregationFunction.java | 22 --
 2 files changed, 10 insertions(+), 15 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments updated (2abd892fe1 -> 4f05fd71ae)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 2abd892fe1 Release all segments of a table in 
releaseAndRemoveAllSegments method
 add 7b69d094be Refactoring the upsert compaction related code (#12275)
 add ced6bc282e Use higher fetch timeout for Kinesis (#12214)
 add f1fec060a6 Support runtime reload for TLS resources (#12277)
 add eeaf1f0811 [feature] allow dim table config to detect/disallow 
duplicate PK  (#12290)
 add 35faeb6712 Revert allowing tag override with upserts (#12311)
 add 63e91ef95f Support table suffix in ZkBasicAuthAccessControlFactory 
(#12310)
 add 91ffcc759f Reduce Disk Footprint for Segment Processor Framework to 
Avoid Out of Disk Issues (#12220)
 add f43664dd13 Support server level consumption throttle (#12292)
 add 9c1bb02dec Misc fixes for upsert metadata manager (#12319)
 add 76d0eb25db Reduce Heap Usage of OnHeapStringDictionary (#12223)
 add 7978d29031 Mark distribution as multi-release (#12300)
 add 5a382f2e7d Shared aggregations in StarTree (#12164)
 add 2acf8ea354 correct errmsg format for Preconditions.check... (#12327)
 add 23dbb08e8d Make server resource classes configurable (#12324)
 add ed6761a982 Fix "rewind()" for CompactedPinotSegmentRecordReader 
(#12329)
 add 4823802886 Wire soft upsert delete for compaction task (#12330)
 add 6cc1915140 Add HttpHeaders in broker event listener requestContext 
(#12258)
 add e1d20d75d5 Making taskManager resources protected for derived classes 
to override in their setUp() method. (#12335)
 add 1a82ba6fc6 Add CPU metrics for minion purge task (#12337)
 add d5014786db Add first implementation of clpMatch that doesn't 
explicitly use indexes. (#12291)
 add 3ab28348fb auto reload TLS resources when loading from local files 
(#12325)
 add 8988b755d6 Remove segments with empty download url in 
UpsertCompactionTask (#12320)
 add dd8be2a229 add TextMatchFilterOptimizer to maximally push down 
`text_match` filters to Lucene (#12339)
 add 17db0fd17b add ScalingThreadPoolExecutor and use for realtime Lucene 
thread pool (#12274)
 add 4f05fd71ae Release all segments of a table in 
releaseAndRemoveAllSegments method

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (2abd892fe1)
\
 N -- N -- N   
refs/heads/stop-all-segments-in-releaseAndRemoveAllSegments (4f05fd71ae)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 LICENSE-binary |   2 +-
 .../broker/ZkBasicAuthAccessControlFactory.java|   3 +-
 .../requesthandler/BaseBrokerRequestHandler.java   |  10 +-
 pinot-common/pom.xml   |   8 +
 .../common/function/TransformFunctionType.java |   3 +
 .../apache/pinot/common/http/MultiHttpRequest.java |  51 +-
 .../pinot/common/metrics/AbstractMetrics.java  |   6 +
 .../apache/pinot/common/metrics/MinionTimer.java   |   2 +-
 ...leSizeInfo.java => ValidDocIdMetadataInfo.java} |  42 +-
 .../common/tier/FixedTierSegmentSelector.java  |   2 +-
 .../common/utils/ScalingThreadPoolExecutor.java| 128 +
 .../org/apache/pinot/common/utils/TlsUtils.java| 258 +++--
 .../pinot/common/utils/grpc/GrpcQueryClient.java   |  24 +-
 .../sql/parsers/rewriter/CLPDecodeRewriter.java| 177 --
 .../pinot/sql/parsers/rewriter/ClpRewriter.java| 633 +
 .../function/FunctionDefinitionRegistryTest.java   |   6 +-
 .../pinot/common/http/MultiHttpRequestTest.java| 106 +++-
 .../pinot/common/utils/FALFInternerTest.java   | 168 ++
 .../utils/ScalingThreadPoolExecutorTest.java   |  69 +++
 .../apache/pinot/common/utils/TlsUtilsTest.java| 286 ++
 .../parsers/rewriter/CLPDecodeRewriterTest.java|  65 ---
 .../sql/parsers/rewriter/ClpRewriterTest.java  | 282 +
 .../src/test/resources/tls/keystore-updated.p12| Bin 0 -> 2581 bytes
 pinot-common/src/test/resources/tls/keystore.p12   | Bin 0 -> 2581 bytes
 .../src/test/resources/tls/truststore-updated.p12  | Bin 0 -> 1186 bytes
 pinot-common/src/test/resources/tls/truststore.p12 | Bin 0 -> 1186 bytes
 .../access/ZkBasicAuthAccessControlFactory.java|   4 +-
 ...

(pinot) branch fix-DistinctCountThetaSketchAggregationFunction created (now 577e5e2c58)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-DistinctCountThetaSketchAggregationFunction
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 577e5e2c58 Fix backward compatible issue in 
DistinctCountThetaSketchAggregationFunction

This branch includes the following new commits:

 new 577e5e2c58 Fix backward compatible issue in 
DistinctCountThetaSketchAggregationFunction

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch fix-DistinctCountThetaSketchAggregationFunction
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 577e5e2c58c26fa16ac7a692c0741ed47033cc63
Author: jlli_LinkedIn 
AuthorDate: Wed Jan 31 19:09:32 2024 -0800

Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction
---
 .../function/DistinctCountThetaSketchAggregationFunction.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
index 83709857f9..34afe70d37 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
@@ -1024,7 +1024,8 @@ public class DistinctCountThetaSketchAggregationFunction
 int numAccumulators = accumulators.size();
 List mergedSketches = new ArrayList<>(numAccumulators);
 
-for (ThetaSketchAccumulator accumulator : accumulators) {
+for (Object accumulatorObject : accumulators) {
+  ThetaSketchAccumulator accumulator = 
convertSketchAccumulator(accumulatorObject);
   accumulator.setThreshold(_accumulatorThreshold);
   accumulator.setSetOperationBuilder(_setOperationBuilder);
   mergedSketches.add(accumulator.getResult());


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch master updated: Add CPU metrics for minion purge task (#12337)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 1a82ba6fc6 Add CPU metrics for minion purge task (#12337)
1a82ba6fc6 is described below

commit 1a82ba6fc6e9ac90f8149bf9432b8cc94eeab8b0
Author: Florence Zhang 
AuthorDate: Mon Jan 29 21:01:02 2024 -0800

Add CPU metrics for minion purge task (#12337)
---
 .../org/apache/pinot/common/metrics/MinionTimer.java  |  2 +-
 .../plugin/minion/tasks/purge/PurgeTaskExecutor.java  | 19 +--
 .../minion/tasks/purge/PurgeTaskExecutorTest.java |  3 ++-
 3 files changed, 20 insertions(+), 4 deletions(-)

diff --git 
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionTimer.java 
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionTimer.java
index ff29a2c367..67d2830c81 100644
--- 
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionTimer.java
+++ 
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/MinionTimer.java
@@ -22,7 +22,7 @@ import org.apache.pinot.common.Utils;
 
 
 public enum MinionTimer implements AbstractMetrics.Timer {
-  TASK_EXECUTION(false), TASK_QUEUEING(false);
+  TASK_EXECUTION(false), TASK_QUEUEING(false), TASK_THREAD_CPU_TIME_NS(false);
 
   private final String _timerName;
   private final boolean _global;
diff --git 
a/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/purge/PurgeTaskExecutor.java
 
b/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/purge/PurgeTaskExecutor.java
index bbc9af3b54..f6b0ca8e39 100644
--- 
a/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/purge/PurgeTaskExecutor.java
+++ 
b/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/purge/PurgeTaskExecutor.java
@@ -19,9 +19,14 @@
 package org.apache.pinot.plugin.minion.tasks.purge;
 
 import java.io.File;
+import java.lang.management.ManagementFactory;
+import java.lang.management.ThreadMXBean;
 import java.util.Collections;
 import java.util.Map;
+import java.util.concurrent.TimeUnit;
 import 
org.apache.pinot.common.metadata.segment.SegmentZKMetadataCustomMapModifier;
+import org.apache.pinot.common.metrics.MinionMetrics;
+import org.apache.pinot.common.metrics.MinionTimer;
 import org.apache.pinot.core.common.MinionConstants;
 import org.apache.pinot.core.minion.PinotTaskConfig;
 import org.apache.pinot.core.minion.SegmentPurger;
@@ -30,13 +35,18 @@ import 
org.apache.pinot.plugin.minion.tasks.SegmentConversionResult;
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.data.Schema;
 import org.apache.pinot.spi.utils.builder.TableNameBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 
 public class PurgeTaskExecutor extends BaseSingleSegmentConversionExecutor {
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(PurgeTaskExecutor.class);
+  protected final MinionMetrics _minionMetrics = MinionMetrics.get();
   public static final String RECORD_PURGER_KEY = "recordPurger";
   public static final String RECORD_MODIFIER_KEY = "recordModifier";
   public static final String NUM_RECORDS_PURGED_KEY = "numRecordsPurged";
   public static final String NUM_RECORDS_MODIFIED_KEY = "numRecordsModified";
+  private static final ThreadMXBean MX_BEAN = 
ManagementFactory.getThreadMXBean();
 
   @Override
   protected SegmentConversionResult convert(PinotTaskConfig pinotTaskConfig, 
File indexDir, File workingDir)
@@ -44,6 +54,7 @@ public class PurgeTaskExecutor extends 
BaseSingleSegmentConversionExecutor {
 Map configs = pinotTaskConfig.getConfigs();
 String tableNameWithType = configs.get(MinionConstants.TABLE_NAME_KEY);
 String rawTableName = 
TableNameBuilder.extractRawTableName(tableNameWithType);
+String taskType = pinotTaskConfig.getTaskType();
 
 SegmentPurger.RecordPurgerFactory recordPurgerFactory = 
MINION_CONTEXT.getRecordPurgerFactory();
 TableConfig tableConfig = getTableConfig(tableNameWithType);
@@ -57,7 +68,11 @@ public class PurgeTaskExecutor extends 
BaseSingleSegmentConversionExecutor {
 _eventObserver.notifyProgress(pinotTaskConfig, "Purging segment: " + 
indexDir);
 SegmentPurger segmentPurger =
 new SegmentPurger(indexDir, workingDir, tableConfig, schema, 
recordPurger, recordModifier);
+long purgeTaskStartTimeNs = MX_BEAN.getCurrentThreadCpuTime();
 File purgedSegmentFile = segmentPurger.purgeSegment();
+long purgeTaskEndTimeNs = MX_BEAN.getCurrentThreadCpuTime();
+_minionMetrics.addTimedTableValue(tableNameWithType, taskType, 
MinionTimer.TASK_THREAD_CPU_TIME_NS,
+purgeTaskEndT

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments updated (3ed9d68147 -> 2abd892fe1)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 3ed9d68147 Release all segments of a table in 
releaseAndRemoveAllSegments method
 add 2abd892fe1 Release all segments of a table in 
releaseAndRemoveAllSegments method

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (3ed9d68147)
\
 N -- N -- N   
refs/heads/stop-all-segments-in-releaseAndRemoveAllSegments (2abd892fe1)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../org/apache/pinot/core/data/manager/BaseTableDataManager.java | 9 +
 1 file changed, 9 insertions(+)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments updated (cb5a82080f -> 3ed9d68147)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard cb5a82080f Release all segments of a table in 
releaseAndRemoveAllSegments method
 add 3ed9d68147 Release all segments of a table in 
releaseAndRemoveAllSegments method

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (cb5a82080f)
\
 N -- N -- N   
refs/heads/stop-all-segments-in-releaseAndRemoveAllSegments (3ed9d68147)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../apache/pinot/core/data/manager/BaseTableDataManager.java   | 10 ++
 1 file changed, 6 insertions(+), 4 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments updated (1a7ae54da8 -> cb5a82080f)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 1a7ae54da8 Release all segments of a table in 
releaseAndRemoveAllSegments method
 new cb5a82080f Release all segments of a table in 
releaseAndRemoveAllSegments method

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (1a7ae54da8)
\
 N -- N -- N   
refs/heads/stop-all-segments-in-releaseAndRemoveAllSegments (cb5a82080f)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/pinot/core/data/manager/BaseTableDataManager.java| 1 -
 1 file changed, 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Release all segments of a table in releaseAndRemoveAllSegments method

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch stop-all-segments-in-releaseAndRemoveAllSegments
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit cb5a82080f6ae909311ce5dcc12c2298e060084d
Author: jlli_LinkedIn 
AuthorDate: Sun Jan 21 16:18:12 2024 -0800

Release all segments of a table in releaseAndRemoveAllSegments method
---
 .../org/apache/pinot/core/data/manager/BaseTableDataManager.java | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
index 119bede805..d5248dc71b 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
@@ -33,6 +33,7 @@ import java.util.Map;
 import java.util.UUID;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
 import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
@@ -223,9 +224,11 @@ public abstract class BaseTableDataManager implements 
TableDataManager {
   segmentDataManagers = new ArrayList<>(_segmentDataManagerMap.values());
   _segmentDataManagerMap.clear();
 }
+ExecutorService stopExecutorService = 
Executors.newFixedThreadPool(segmentDataManagers.size());
 for (SegmentDataManager segmentDataManager : segmentDataManagers) {
-  releaseSegment(segmentDataManager);
+  stopExecutorService.submit(() -> releaseSegment(segmentDataManager));
 }
+stopExecutorService.shutdown();
   }
 
   @Override


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Release all segments of a table in releaseAndRemoveAllSegments method

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch stop-all-segments-in-releaseAndRemoveAllSegments
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 1a7ae54da8fec3bdf0f1f794c0492dd8c29a
Author: jlli_LinkedIn 
AuthorDate: Sun Jan 21 16:18:12 2024 -0800

Release all segments of a table in releaseAndRemoveAllSegments method
---
 .../org/apache/pinot/core/data/manager/BaseTableDataManager.java| 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git 
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
 
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
index 119bede805..5750c919b3 100644
--- 
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
+++ 
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
@@ -32,7 +32,9 @@ import java.util.List;
 import java.util.Map;
 import java.util.UUID;
 import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executor;
 import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
 import java.util.concurrent.Semaphore;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicInteger;
@@ -223,9 +225,11 @@ public abstract class BaseTableDataManager implements 
TableDataManager {
   segmentDataManagers = new ArrayList<>(_segmentDataManagerMap.values());
   _segmentDataManagerMap.clear();
 }
+ExecutorService stopExecutorService = 
Executors.newFixedThreadPool(segmentDataManagers.size());
 for (SegmentDataManager segmentDataManager : segmentDataManagers) {
-  releaseSegment(segmentDataManager);
+  stopExecutorService.submit(() -> releaseSegment(segmentDataManager));
 }
+stopExecutorService.shutdown();
   }
 
   @Override


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch stop-all-segments-in-releaseAndRemoveAllSegments created (now 1a7ae54da8)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch stop-all-segments-in-releaseAndRemoveAllSegments
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 1a7ae54da8 Release all segments of a table in 
releaseAndRemoveAllSegments method

This branch includes the following new commits:

 new 1a7ae54da8 Release all segments of a table in 
releaseAndRemoveAllSegments method

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch master updated: Backwards compatible theta sketch aggregation (#12288)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new 5cc7231dc0 Backwards compatible theta sketch aggregation (#12288)
5cc7231dc0 is described below

commit 5cc7231dc0fc52d56b2a5435ba30489799013785
Author: David Cromberge 
AuthorDate: Sun Jan 21 22:42:08 2024 +

Backwards compatible theta sketch aggregation (#12288)

* Backwards compatible theta sketch aggregation

Servers running on versions before upgrading Pinot to the 
ThetaSketchAccumulator
would return Sketches directly to the merge function.  This ensures that 
there
is backwards compatibility between the two.

* Add Theta Sketch distinct count queries to compatibility check queries
---
 .../config/queries/feature-test-1-sql.queries  |  6 +++---
 .../config/queries/feature-test-2-sql-realtime.queries |  4 ++--
 .../query-results/feature-test-1-rest-sql.results  |  6 +++---
 .../query-results/feature-test-2-sql-realtime.results  |  4 ++--
 .../DistinctCountThetaSketchAggregationFunction.java   | 18 --
 5 files changed, 26 insertions(+), 12 deletions(-)

diff --git 
a/compatibility-verifier/sample-test-suite/config/queries/feature-test-1-sql.queries
 
b/compatibility-verifier/sample-test-suite/config/queries/feature-test-1-sql.queries
index 37a6120a5d..38b8484243 100644
--- 
a/compatibility-verifier/sample-test-suite/config/queries/feature-test-1-sql.queries
+++ 
b/compatibility-verifier/sample-test-suite/config/queries/feature-test-1-sql.queries
@@ -22,7 +22,7 @@ SELECT count(*) FROM FeatureTest1 WHERE generationNumber = 
__GENERATION_NUMBER__
 SELECT sum(intMetric1), sumMV(intDimMV1), min(intMetric1), minMV(intDimMV2), 
max(longDimSV1), maxMV(intDimMV1) FROM FeatureTest1 WHERE generationNumber = 
__GENERATION_NUMBER__
 SELECT count(longDimSV1), countMV(intDimMV1), avg(floatMetric1), 
avgMV(intDimMV2), minMaxRange(doubleMetric1), minMaxRangeMV(intDimMV2) FROM 
FeatureTest1 WHERE generationNumber = __GENERATION_NUMBER__
 SELECT percentile(longDimSV1, 80), percentileMV(intDimMV1, 90), 
percentileEst(longDimSV1, 80), percentileEstMV(intDimMV1, 90), 
percentileTDigest(longDimSV1, 80), percentileTDigestMV(intDimMV1, 90) FROM 
FeatureTest1 WHERE generationNumber = __GENERATION_NUMBER__
-SELECT distinctCount(longDimSV1), distinctCountMV(intDimMV1), 
distinctCountHLL(longDimSV1), distinctCountHLLMV(intDimMV1) FROM FeatureTest1 
WHERE generationNumber = __GENERATION_NUMBER__
+SELECT distinctCount(longDimSV1), distinctCountMV(intDimMV1), 
distinctCountHLL(longDimSV1), distinctCountHLLMV(intDimMV1), 
distinctCountThetaSketch(longDimSV1) FROM FeatureTest1 WHERE generationNumber = 
__GENERATION_NUMBER__
 
 # Selection
 SELECT longDimSV2, stringDimSV1, textDim1, bytesDimSV1 FROM FeatureTest1 WHERE 
generationNumber = __GENERATION_NUMBER__ ORDER BY longDimSV2 LIMIT 9
@@ -46,14 +46,14 @@ SELECT longDimSV1, intDimMV1, count(*) FROM FeatureTest1 
WHERE generationNumber
 SELECT longDimSV1, intDimMV1, sum(intMetric1), sumMV(intDimMV1), 
min(intMetric1), minMV(intDimMV2), max(longDimSV1), maxMV(intDimMV1) FROM 
FeatureTest1 WHERE generationNumber = __GENERATION_NUMBER__ GROUP BY 
longDimSV1, intDimMV1 ORDER BY longDimSV1 LIMIT 5
 SELECT longDimSV1, intDimMV1, count(longDimSV1), countMV(intDimMV1), 
avg(floatMetric1), avgMV(intDimMV2), minMaxRange(doubleMetric1), 
minMaxRangeMV(intDimMV2) FROM FeatureTest1 WHERE generationNumber = 
__GENERATION_NUMBER__ GROUP BY longDimSV1, intDimMV1 ORDER BY longDimSV1 LIMIT 5
 SELECT longDimSV1, intDimMV1, percentile(longDimSV1, 80), 
percentileMV(intDimMV1, 90), percentileEst(longDimSV1, 80), 
percentileEstMV(intDimMV1, 90), percentileTDigest(longDimSV1, 80), 
percentileTDigestMV(intDimMV1, 90) FROM FeatureTest1 WHERE generationNumber = 
__GENERATION_NUMBER__ GROUP BY longDimSV1, intDimMV1 ORDER BY longDimSV1 LIMIT 5
-SELECT longDimSV1, intDimMV1, distinctCount(longDimSV1), 
distinctCountMV(intDimMV1), distinctCountHLL(longDimSV1), 
distinctCountHLLMV(intDimMV1) FROM FeatureTest1 WHERE generationNumber = 
__GENERATION_NUMBER__ GROUP BY longDimSV1, intDimMV1 ORDER BY longDimSV1 LIMIT 5
+SELECT longDimSV1, intDimMV1, distinctCount(longDimSV1), 
distinctCountMV(intDimMV1), distinctCountHLL(longDimSV1), 
distinctCountHLLMV(intDimMV1), distinctCountThetaSketch(longDimSV1) FROM 
FeatureTest1 WHERE generationNumber = __GENERATION_NUMBER__ GROUP BY 
longDimSV1, intDimMV1 ORDER BY longDimSV1 LIMIT 5
 
 # Selection & Filtering & Grouping on Aggregation
 SELECT longDimSV1, intDimMV1, count(*) FROM FeatureTest1 WHERE 
generationNumber = __GENERATION_NUMBER__ AND (stringDimSV1 != 's1-6' AND 
longDimSV1 BETWEEN 10 AND 1000 OR (intDimMV1 < 42 AND stringDimMV2 IN 
('m2-0-0', 'm2-2-0') AND intDimMV2 NOT IN (6,72))) GROUP BY longDimSV1, 
intDimMV1 ORDER BY longDimSV1, intDimMV1 LIMI

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (83911abc93 -> 08efc8c2f1)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


from 83911abc93 Address PR comments
 add 08efc8c2f1 Add logic to consider the case when instances are moved 
across pools

No new revisions were added by this update.

Summary of changes:
 .../InstanceReplicaGroupPartitionSelector.java | 75 +
 .../instance/InstanceTagPoolSelector.java  | 38 ++-
 .../InstanceReplicaGroupPartitionSelectorTest.java | 76 --
 .../java/org/apache/pinot/spi/utils/Pairs.java | 23 ++-
 4 files changed, 171 insertions(+), 41 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (ff4b642ee6 -> 83911abc93)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard ff4b642ee6 Address PR comments
 add 83911abc93 Address PR comments

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (ff4b642ee6)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (83911abc93)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../core/assignment/instance/InstanceTagPoolSelector.java  | 10 --
 1 file changed, 4 insertions(+), 6 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated: Address PR comments

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


The following commit(s) were added to 
refs/heads/maintain-pool-selection-for-minimizeDataMovement by this push:
 new ff4b642ee6 Address PR comments
ff4b642ee6 is described below

commit ff4b642ee60d0295d3efb9ed64009b9b8378845c
Author: jlli_LinkedIn 
AuthorDate: Thu Nov 16 21:30:54 2023 -0800

Address PR comments
---
 .../core/assignment/instance/InstanceTagPoolSelector.java  | 14 +++---
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceTagPoolSelector.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceTagPoolSelector.java
index 755e7aa713..b40f094e1d 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceTagPoolSelector.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceTagPoolSelector.java
@@ -22,11 +22,11 @@ import com.google.common.base.Preconditions;
 import java.util.ArrayList;
 import java.util.Comparator;
 import java.util.HashMap;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
 import java.util.TreeMap;
+import java.util.TreeSet;
 import javax.annotation.Nullable;
 import org.apache.helix.model.InstanceConfig;
 import org.apache.pinot.common.assignment.InstancePartitions;
@@ -50,8 +50,7 @@ public class InstanceTagPoolSelector {
   private final InstancePartitions _existingInstancePartitions;
 
   public InstanceTagPoolSelector(InstanceTagPoolConfig tagPoolConfig, String 
tableNameWithType,
-  boolean minimizeDataMovement,
-  @Nullable InstancePartitions existingInstancePartitions) {
+  boolean minimizeDataMovement, @Nullable InstancePartitions 
existingInstancePartitions) {
 _tagPoolConfig = tagPoolConfig;
 _tableNameWithType = tableNameWithType;
 _minimizeDataMovement = minimizeDataMovement;
@@ -124,7 +123,7 @@ public class InstanceTagPoolSelector {
 
 poolsToSelect = new ArrayList<>(numPoolsToSelect);
 if (_minimizeDataMovement && _existingInstancePartitions != null) {
-  Set existingPools = new HashSet<>(numPoolsToSelect);
+  Set existingPools = new TreeSet<>();
   // Keep the same pool if it's already been used for the table.
   int existingNumPartitions = 
_existingInstancePartitions.getNumPartitions();
   int existingNumReplicaGroups = 
_existingInstancePartitions.getNumReplicaGroups();
@@ -135,9 +134,10 @@ public class InstanceTagPoolSelector {
   List existingInstances = 
_existingInstancePartitions.getInstances(partitionId, replicaGroupId);
   for (String existingInstance : existingInstances) {
 Integer existingPool = instanceToPoolMap.get(existingInstance);
-if (existingPool != null & pools.contains(existingPool)) {
-  poolsToSelect.add(existingPool);
-  existingPools.add(existingPool);
+if (existingPool != null) {
+  if (existingPools.add(existingPool)) {
+poolsToSelect.add(existingPool);
+  }
   foundExistingPoolForReplicaGroup = true;
   break;
 }


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (bb38609a00 -> d9ae2e9432)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


omit bb38609a00 Update 
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstancePartitionSelector.java
 add d9ae2e9432 Update 
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstancePartitionSelector.java

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (bb38609a00)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (d9ae2e9432)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../helix/core/assignment/instance/InstancePartitionSelector.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (95ed621750 -> bb38609a00)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


from 95ed621750 Enhance the minimizeDataMovement to keep the existing pool 
assignment
 add bb38609a00 Update 
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstancePartitionSelector.java

No new revisions were added by this update.

Summary of changes:
 .../helix/core/assignment/instance/InstancePartitionSelector.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (993aa6b71c -> 95ed621750)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


omit 993aa6b71c Enhance the minimizeDataMovement to keep the existing pool 
assignment
 add 95ed621750 Enhance the minimizeDataMovement to keep the existing pool 
assignment

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (993aa6b71c)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (95ed621750)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../helix/core/assignment/instance/InstanceTagPoolSelector.java | 2 --
 1 file changed, 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (3a3c49386a -> 993aa6b71c)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


omit 3a3c49386a Enhance the minimizeDataMovement to keep the existing pool 
assignment
 add 993aa6b71c Enhance the minimizeDataMovement to keep the existing pool 
assignment

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (3a3c49386a)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (993aa6b71c)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../instance/InstanceTagPoolSelector.java   | 21 -
 .../InstanceReplicaGroupPartitionSelectorTest.java  |  2 +-
 2 files changed, 17 insertions(+), 6 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (50cda350c0 -> 3a3c49386a)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


omit 50cda350c0 Enhance the minimizeDataMovement to keep the existing pool 
assignment
 add 3a3c49386a Enhance the minimizeDataMovement to keep the existing pool 
assignment

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (50cda350c0)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (3a3c49386a)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../helix/core/assignment/instance/InstanceTagPoolSelector.java  | 1 -
 1 file changed, 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (50e0c459e6 -> 50cda350c0)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 50e0c459e6 Enhance the minimizeDataMovement to keep the existing pool 
assignment
 add 50cda350c0 Enhance the minimizeDataMovement to keep the existing pool 
assignment

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (50e0c459e6)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (50cda350c0)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../assignment/InstanceAssignmentConfigUtils.java  |  2 +-
 .../common/utils/config/TableConfigSerDeTest.java  |  2 +-
 .../instance/FDAwareInstancePartitionSelector.java |  6 +-
 .../instance/InstanceAssignmentDriver.java | 10 +--
 .../instance/InstancePartitionSelector.java|  4 +-
 .../instance/InstancePartitionSelectorFactory.java | 18 ++---
 .../InstanceReplicaGroupPartitionSelector.java | 10 +--
 .../instance/InstanceTagPoolSelector.java  |  8 +-
 .../MirrorServerSetInstancePartitionSelector.java  |  4 +-
 ...anceAssignmentRestletResourceStatelessTest.java |  6 +-
 .../instance/InstanceAssignmentTest.java   | 94 --
 .../TableRebalancerClusterStatelessTest.java   |  4 +-
 .../table/assignment/InstanceAssignmentConfig.java | 16 ++--
 .../InstanceReplicaGroupPartitionConfig.java   |  2 +
 14 files changed, 96 insertions(+), 90 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Enhance the minimizeDataMovement to keep the existing pool assignment

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 50e0c459e65094e622e6a05fb93bc8d33f69d6a9
Author: jlli_LinkedIn 
AuthorDate: Sun Nov 5 10:55:11 2023 -0800

Enhance the minimizeDataMovement to keep the existing pool assignment
---
 .../instance/InstanceAssignmentDriver.java |  12 ++-
 .../InstanceReplicaGroupPartitionSelector.java |  70 --
 .../instance/InstanceTagPoolSelector.java  |  55 +--
 .../instance/InstanceAssignmentTest.java   | 103 +
 4 files changed, 219 insertions(+), 21 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
index 6d869b86c1..ffde3fce1e 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
@@ -30,6 +30,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceAssignmentConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceConstraintConfig;
 import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import 
org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
 import org.apache.pinot.spi.utils.builder.TableNameBuilder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -64,8 +65,8 @@ public class InstanceAssignmentDriver {
   }
 
   public InstancePartitions assignInstances(InstancePartitionsType 
instancePartitionsType,
-  List instanceConfigs, @Nullable InstancePartitions 
existingInstancePartitions, @Nullable
-  InstancePartitions preConfiguredInstancePartitions) {
+  List instanceConfigs, @Nullable InstancePartitions 
existingInstancePartitions,
+  @Nullable InstancePartitions preConfiguredInstancePartitions) {
 String tableNameWithType = _tableConfig.getTableName();
 InstanceAssignmentConfig assignmentConfig =
 
InstanceAssignmentConfigUtils.getInstanceAssignmentConfig(_tableConfig, 
instancePartitionsType);
@@ -88,8 +89,11 @@ public class InstanceAssignmentDriver {
 String tableNameWithType = _tableConfig.getTableName();
 LOGGER.info("Starting {} instance assignment for table {}", 
instancePartitionsName, tableNameWithType);
 
+InstanceReplicaGroupPartitionConfig instanceReplicaGroupPartitionConfig =
+instanceAssignmentConfig.getReplicaGroupPartitionConfig();
 InstanceTagPoolSelector tagPoolSelector =
-new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType);
+new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType,
+instanceReplicaGroupPartitionConfig, existingInstancePartitions);
 Map> poolToInstanceConfigsMap = 
tagPoolSelector.selectInstances(instanceConfigs);
 
 InstanceConstraintConfig constraintConfig = 
instanceAssignmentConfig.getConstraintConfig();
@@ -105,7 +109,7 @@ public class InstanceAssignmentDriver {
 
 InstancePartitionSelector instancePartitionSelector =
 
InstancePartitionSelectorFactory.getInstance(instanceAssignmentConfig.getPartitionSelector(),
-instanceAssignmentConfig.getReplicaGroupPartitionConfig(), 
tableNameWithType, existingInstancePartitions,
+instanceReplicaGroupPartitionConfig, tableNameWithType, 
existingInstancePartitions,
 preConfiguredInstancePartitions);
 InstancePartitions instancePartitions = new 
InstancePartitions(instancePartitionsName);
 instancePartitionSelector.selectInstances(poolToInstanceConfigsMap, 
instancePartitions);
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
index de1e681d17..d7b5084f2b 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
@@ -22,18 +22,21 @@ import com.google.common.base.Preconditions;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.Deque;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.Iterator;
 import java.util.LinkedHashSet;
 import java.util.LinkedList;
 import j

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (1a8c7a1433 -> 50e0c459e6)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 1a8c7a1433 Enhance the minimizeDataMovement to keep the existing pool 
assignment
 new 50e0c459e6 Enhance the minimizeDataMovement to keep the existing pool 
assignment

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (1a8c7a1433)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (50e0c459e6)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/assignment/instance/InstanceReplicaGroupPartitionSelector.java  | 1 -
 1 file changed, 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (7cc4f32a0a -> 1a8c7a1433)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 7cc4f32a0a Enhance the minimizeDataMovement to keep the existing pool 
assignment
 new 1a8c7a1433 Enhance the minimizeDataMovement to keep the existing pool 
assignment

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (7cc4f32a0a)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (1a8c7a1433)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../helix/core/assignment/instance/InstanceAssignmentDriver.java | 4 ++--
 .../assignment/instance/InstanceReplicaGroupPartitionSelector.java   | 5 ++---
 .../helix/core/assignment/instance/InstanceTagPoolSelector.java  | 5 ++---
 3 files changed, 6 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Enhance the minimizeDataMovement to keep the existing pool assignment

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 1a8c7a14330c835f2e232febe0ebf5c3dacd6472
Author: jlli_LinkedIn 
AuthorDate: Sun Nov 5 10:55:11 2023 -0800

Enhance the minimizeDataMovement to keep the existing pool assignment
---
 .../instance/InstanceAssignmentDriver.java |  12 ++-
 .../InstanceReplicaGroupPartitionSelector.java |  71 --
 .../instance/InstanceTagPoolSelector.java  |  55 +--
 .../instance/InstanceAssignmentTest.java   | 103 +
 4 files changed, 220 insertions(+), 21 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
index 6d869b86c1..ffde3fce1e 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
@@ -30,6 +30,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceAssignmentConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceConstraintConfig;
 import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import 
org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
 import org.apache.pinot.spi.utils.builder.TableNameBuilder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -64,8 +65,8 @@ public class InstanceAssignmentDriver {
   }
 
   public InstancePartitions assignInstances(InstancePartitionsType 
instancePartitionsType,
-  List instanceConfigs, @Nullable InstancePartitions 
existingInstancePartitions, @Nullable
-  InstancePartitions preConfiguredInstancePartitions) {
+  List instanceConfigs, @Nullable InstancePartitions 
existingInstancePartitions,
+  @Nullable InstancePartitions preConfiguredInstancePartitions) {
 String tableNameWithType = _tableConfig.getTableName();
 InstanceAssignmentConfig assignmentConfig =
 
InstanceAssignmentConfigUtils.getInstanceAssignmentConfig(_tableConfig, 
instancePartitionsType);
@@ -88,8 +89,11 @@ public class InstanceAssignmentDriver {
 String tableNameWithType = _tableConfig.getTableName();
 LOGGER.info("Starting {} instance assignment for table {}", 
instancePartitionsName, tableNameWithType);
 
+InstanceReplicaGroupPartitionConfig instanceReplicaGroupPartitionConfig =
+instanceAssignmentConfig.getReplicaGroupPartitionConfig();
 InstanceTagPoolSelector tagPoolSelector =
-new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType);
+new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType,
+instanceReplicaGroupPartitionConfig, existingInstancePartitions);
 Map> poolToInstanceConfigsMap = 
tagPoolSelector.selectInstances(instanceConfigs);
 
 InstanceConstraintConfig constraintConfig = 
instanceAssignmentConfig.getConstraintConfig();
@@ -105,7 +109,7 @@ public class InstanceAssignmentDriver {
 
 InstancePartitionSelector instancePartitionSelector =
 
InstancePartitionSelectorFactory.getInstance(instanceAssignmentConfig.getPartitionSelector(),
-instanceAssignmentConfig.getReplicaGroupPartitionConfig(), 
tableNameWithType, existingInstancePartitions,
+instanceReplicaGroupPartitionConfig, tableNameWithType, 
existingInstancePartitions,
 preConfiguredInstancePartitions);
 InstancePartitions instancePartitions = new 
InstancePartitions(instancePartitionsName);
 instancePartitionSelector.selectInstances(poolToInstanceConfigsMap, 
instancePartitions);
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
index de1e681d17..e0008eab80 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
@@ -21,19 +21,23 @@ package 
org.apache.pinot.controller.helix.core.assignment.instance;
 import com.google.common.base.Preconditions;
 import java.util.ArrayList;
 import java.util.Collections;
+import java.util.Comparator;
 import java.util.Deque;
+import java.util.HashMap;
 import java.util.Has

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (0704cce453 -> 7cc4f32a0a)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard 0704cce453 Enhance the minimizeDataMovement to keep the existing pool 
assignment
 new 7cc4f32a0a Enhance the minimizeDataMovement to keep the existing pool 
assignment

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (0704cce453)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (7cc4f32a0a)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../helix/core/assignment/instance/InstanceAssignmentTest.java   | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Enhance the minimizeDataMovement to keep the existing pool assignment

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 7cc4f32a0a37a75a590b0ee62e5ac7ada3cafbe7
Author: jlli_LinkedIn 
AuthorDate: Sun Nov 5 10:55:11 2023 -0800

Enhance the minimizeDataMovement to keep the existing pool assignment
---
 .../instance/InstanceAssignmentDriver.java |   8 +-
 .../InstanceReplicaGroupPartitionSelector.java |  70 --
 .../instance/InstanceTagPoolSelector.java  |  50 +-
 .../instance/InstanceAssignmentTest.java   | 103 +
 4 files changed, 216 insertions(+), 15 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
index 6d869b86c1..6b833a436a 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
@@ -30,6 +30,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceAssignmentConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceConstraintConfig;
 import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import 
org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
 import org.apache.pinot.spi.utils.builder.TableNameBuilder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -88,8 +89,11 @@ public class InstanceAssignmentDriver {
 String tableNameWithType = _tableConfig.getTableName();
 LOGGER.info("Starting {} instance assignment for table {}", 
instancePartitionsName, tableNameWithType);
 
+InstanceReplicaGroupPartitionConfig instanceReplicaGroupPartitionConfig =
+instanceAssignmentConfig.getReplicaGroupPartitionConfig();
 InstanceTagPoolSelector tagPoolSelector =
-new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType);
+new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType,
+instanceReplicaGroupPartitionConfig, existingInstancePartitions);
 Map> poolToInstanceConfigsMap = 
tagPoolSelector.selectInstances(instanceConfigs);
 
 InstanceConstraintConfig constraintConfig = 
instanceAssignmentConfig.getConstraintConfig();
@@ -105,7 +109,7 @@ public class InstanceAssignmentDriver {
 
 InstancePartitionSelector instancePartitionSelector =
 
InstancePartitionSelectorFactory.getInstance(instanceAssignmentConfig.getPartitionSelector(),
-instanceAssignmentConfig.getReplicaGroupPartitionConfig(), 
tableNameWithType, existingInstancePartitions,
+instanceReplicaGroupPartitionConfig, tableNameWithType, 
existingInstancePartitions,
 preConfiguredInstancePartitions);
 InstancePartitions instancePartitions = new 
InstancePartitions(instancePartitionsName);
 instancePartitionSelector.selectInstances(poolToInstanceConfigsMap, 
instancePartitions);
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
index de1e681d17..2e639409bc 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
@@ -21,19 +21,23 @@ package 
org.apache.pinot.controller.helix.core.assignment.instance;
 import com.google.common.base.Preconditions;
 import java.util.ArrayList;
 import java.util.Collections;
+import java.util.Comparator;
 import java.util.Deque;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.Iterator;
 import java.util.LinkedHashSet;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.PriorityQueue;
 import java.util.Set;
 import java.util.TreeMap;
 import javax.annotation.Nullable;
 import org.apache.helix.model.InstanceConfig;
 import org.apache.pinot.common.assignment.InstancePartitions;
 import 
org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
+import org.apache.pinot.spi.utils.Pairs;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -73,16 +77,66 @@ public class InstanceReplicaGroupPartitionSelector extends 
InstancePartitionSele
   Map> poolToReplicaG

(pinot) 01/01: Enhance the minimizeDataMovement to keep the existing pool assignment

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 0704cce4535e0d0406d8040a0e5a9f87f2e257a7
Author: jlli_LinkedIn 
AuthorDate: Sun Nov 5 10:55:11 2023 -0800

Enhance the minimizeDataMovement to keep the existing pool assignment
---
 .../instance/InstanceAssignmentDriver.java |   8 +-
 .../InstanceReplicaGroupPartitionSelector.java |  70 --
 .../instance/InstanceTagPoolSelector.java  |  50 +-
 .../instance/InstanceAssignmentTest.java   | 102 +
 4 files changed, 215 insertions(+), 15 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
index 6d869b86c1..6b833a436a 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
@@ -30,6 +30,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceAssignmentConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceConstraintConfig;
 import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import 
org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
 import org.apache.pinot.spi.utils.builder.TableNameBuilder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -88,8 +89,11 @@ public class InstanceAssignmentDriver {
 String tableNameWithType = _tableConfig.getTableName();
 LOGGER.info("Starting {} instance assignment for table {}", 
instancePartitionsName, tableNameWithType);
 
+InstanceReplicaGroupPartitionConfig instanceReplicaGroupPartitionConfig =
+instanceAssignmentConfig.getReplicaGroupPartitionConfig();
 InstanceTagPoolSelector tagPoolSelector =
-new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType);
+new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType,
+instanceReplicaGroupPartitionConfig, existingInstancePartitions);
 Map> poolToInstanceConfigsMap = 
tagPoolSelector.selectInstances(instanceConfigs);
 
 InstanceConstraintConfig constraintConfig = 
instanceAssignmentConfig.getConstraintConfig();
@@ -105,7 +109,7 @@ public class InstanceAssignmentDriver {
 
 InstancePartitionSelector instancePartitionSelector =
 
InstancePartitionSelectorFactory.getInstance(instanceAssignmentConfig.getPartitionSelector(),
-instanceAssignmentConfig.getReplicaGroupPartitionConfig(), 
tableNameWithType, existingInstancePartitions,
+instanceReplicaGroupPartitionConfig, tableNameWithType, 
existingInstancePartitions,
 preConfiguredInstancePartitions);
 InstancePartitions instancePartitions = new 
InstancePartitions(instancePartitionsName);
 instancePartitionSelector.selectInstances(poolToInstanceConfigsMap, 
instancePartitions);
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
index de1e681d17..2e639409bc 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
@@ -21,19 +21,23 @@ package 
org.apache.pinot.controller.helix.core.assignment.instance;
 import com.google.common.base.Preconditions;
 import java.util.ArrayList;
 import java.util.Collections;
+import java.util.Comparator;
 import java.util.Deque;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.Iterator;
 import java.util.LinkedHashSet;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.PriorityQueue;
 import java.util.Set;
 import java.util.TreeMap;
 import javax.annotation.Nullable;
 import org.apache.helix.model.InstanceConfig;
 import org.apache.pinot.common.assignment.InstancePartitions;
 import 
org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
+import org.apache.pinot.spi.utils.Pairs;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -73,16 +77,66 @@ public class InstanceReplicaGroupPartitionSelector extends 
InstancePartitionSele
   Map> poolToReplicaG

(pinot) branch maintain-pool-selection-for-minimizeDataMovement updated (f372544ca2 -> 0704cce453)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


 discard f372544ca2 Enhance the minimizeDataMovement to keep the existing pool 
assignment
 new 0704cce453 Enhance the minimizeDataMovement to keep the existing pool 
assignment

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (f372544ca2)
\
 N -- N -- N   
refs/heads/maintain-pool-selection-for-minimizeDataMovement (0704cce453)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../helix/core/assignment/instance/InstanceAssignmentTest.java  | 6 --
 1 file changed, 6 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch maintain-pool-selection-for-minimizeDataMovement created (now f372544ca2)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at f372544ca2 Enhance the minimizeDataMovement to keep the existing pool 
assignment

This branch includes the following new commits:

 new f372544ca2 Enhance the minimizeDataMovement to keep the existing pool 
assignment

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Enhance the minimizeDataMovement to keep the existing pool assignment

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch maintain-pool-selection-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit f372544ca2292c629763dd56dc54bd165ea555f5
Author: jlli_LinkedIn 
AuthorDate: Sun Nov 5 10:55:11 2023 -0800

Enhance the minimizeDataMovement to keep the existing pool assignment
---
 .../instance/InstanceAssignmentDriver.java |   8 +-
 .../InstanceReplicaGroupPartitionSelector.java |  70 +++--
 .../instance/InstanceTagPoolSelector.java  |  50 +-
 .../instance/InstanceAssignmentTest.java   | 108 +
 4 files changed, 221 insertions(+), 15 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
index 6d869b86c1..6b833a436a 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceAssignmentDriver.java
@@ -30,6 +30,7 @@ import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceAssignmentConfig;
 import org.apache.pinot.spi.config.table.assignment.InstanceConstraintConfig;
 import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import 
org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
 import org.apache.pinot.spi.utils.builder.TableNameBuilder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -88,8 +89,11 @@ public class InstanceAssignmentDriver {
 String tableNameWithType = _tableConfig.getTableName();
 LOGGER.info("Starting {} instance assignment for table {}", 
instancePartitionsName, tableNameWithType);
 
+InstanceReplicaGroupPartitionConfig instanceReplicaGroupPartitionConfig =
+instanceAssignmentConfig.getReplicaGroupPartitionConfig();
 InstanceTagPoolSelector tagPoolSelector =
-new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType);
+new 
InstanceTagPoolSelector(instanceAssignmentConfig.getTagPoolConfig(), 
tableNameWithType,
+instanceReplicaGroupPartitionConfig, existingInstancePartitions);
 Map> poolToInstanceConfigsMap = 
tagPoolSelector.selectInstances(instanceConfigs);
 
 InstanceConstraintConfig constraintConfig = 
instanceAssignmentConfig.getConstraintConfig();
@@ -105,7 +109,7 @@ public class InstanceAssignmentDriver {
 
 InstancePartitionSelector instancePartitionSelector =
 
InstancePartitionSelectorFactory.getInstance(instanceAssignmentConfig.getPartitionSelector(),
-instanceAssignmentConfig.getReplicaGroupPartitionConfig(), 
tableNameWithType, existingInstancePartitions,
+instanceReplicaGroupPartitionConfig, tableNameWithType, 
existingInstancePartitions,
 preConfiguredInstancePartitions);
 InstancePartitions instancePartitions = new 
InstancePartitions(instancePartitionsName);
 instancePartitionSelector.selectInstances(poolToInstanceConfigsMap, 
instancePartitions);
diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
index de1e681d17..2e639409bc 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
@@ -21,19 +21,23 @@ package 
org.apache.pinot.controller.helix.core.assignment.instance;
 import com.google.common.base.Preconditions;
 import java.util.ArrayList;
 import java.util.Collections;
+import java.util.Comparator;
 import java.util.Deque;
+import java.util.HashMap;
 import java.util.HashSet;
 import java.util.Iterator;
 import java.util.LinkedHashSet;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
+import java.util.PriorityQueue;
 import java.util.Set;
 import java.util.TreeMap;
 import javax.annotation.Nullable;
 import org.apache.helix.model.InstanceConfig;
 import org.apache.pinot.common.assignment.InstancePartitions;
 import 
org.apache.pinot.spi.config.table.assignment.InstanceReplicaGroupPartitionConfig;
+import org.apache.pinot.spi.utils.Pairs;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -73,16 +77,66 @@ public class InstanceReplicaGroupPartitionSelector extends 
InstancePartitionSele
   Map> poolToReplicaG

(pinot) branch fix-npe-for-minimizeDataMovement deleted (was 1841b566df)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-npe-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


 was 1841b566df Fix the NPE in minimizeDataMovement instance assignment 
strategy

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Fix the NPE in minimizeDataMovement instance assignment strategy

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch fix-npe-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 1841b566df7408dd33b5e48ff041a741b9d22d4e
Author: jlli_LinkedIn 
AuthorDate: Fri Nov 3 15:48:31 2023 -0700

Fix the NPE in minimizeDataMovement instance assignment strategy
---
 .../InstanceReplicaGroupPartitionSelector.java |  11 ++-
 .../InstanceReplicaGroupPartitionSelectorTest.java | 104 +
 2 files changed, 110 insertions(+), 5 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
index b6c62ac12e..de1e681d17 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
@@ -72,11 +72,17 @@ public class InstanceReplicaGroupPartitionSelector extends 
InstancePartitionSele
   Preconditions.checkState(numReplicaGroups > 0, "Number of replica-groups 
must be positive");
   Map> poolToReplicaGroupIdsMap = new TreeMap<>();
   Map replicaGroupIdToPoolMap = new TreeMap<>();
+  Map> poolToCandidateInstancesMap = new TreeMap<>();
   for (int replicaId = 0; replicaId < numReplicaGroups; replicaId++) {
 // Pick one pool for each replica-group based on the table name hash
 int pool = pools.get((tableNameHash + replicaId) % numPools);
 poolToReplicaGroupIdsMap.computeIfAbsent(pool, k -> new 
ArrayList<>()).add(replicaId);
 replicaGroupIdToPoolMap.put(replicaId, pool);
+
+Set candidateInstances =
+poolToCandidateInstancesMap.computeIfAbsent(pool, k -> new 
LinkedHashSet<>());
+List instanceConfigsInPool = 
poolToInstanceConfigsMap.get(pool);
+instanceConfigsInPool.forEach(k -> 
candidateInstances.add(k.getInstanceName()));
   }
   LOGGER.info("Selecting {} replica-groups from pool: {} for table: {}", 
numReplicaGroups, poolToReplicaGroupIdsMap,
   _tableNameWithType);
@@ -132,7 +138,6 @@ public class InstanceReplicaGroupPartitionSelector extends 
InstancePartitionSele
 int existingNumReplicaGroups = 
_existingInstancePartitions.getNumReplicaGroups();
 int numCommonReplicaGroups = Math.min(numReplicaGroups, 
existingNumReplicaGroups);
 
-Map> poolToCandidateInstancesMap = new 
TreeMap<>();
 Map> replicaGroupIdToExistingInstancesMap = new 
TreeMap<>();
 // Step 1: find out the replica groups and their existing instances,
 //   so that these instances can be filtered out and won't be chosen 
for the other replica group.
@@ -142,10 +147,6 @@ public class InstanceReplicaGroupPartitionSelector extends 
InstancePartitionSele
 // Skip the replica group if it's no longer needed.
 continue;
   }
-  Set candidateInstances =
-  poolToCandidateInstancesMap.computeIfAbsent(pool, k -> new 
LinkedHashSet<>());
-  List instanceConfigsInPool = 
poolToInstanceConfigsMap.get(pool);
-  instanceConfigsInPool.forEach(k -> 
candidateInstances.add(k.getInstanceName()));
 
   for (int partitionId = 0; partitionId < existingNumPartitions; 
partitionId++) {
 List existingInstances = 
_existingInstancePartitions.getInstances(partitionId, replicaGroupId);
diff --git 
a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelectorTest.java
 
b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelectorTest.java
new file mode 100644
index 00..889206437f
--- /dev/null
+++ 
b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelectorTest.java
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT W

(pinot) branch fix-npe-for-minimizeDataMovement updated (8315d894b5 -> 1841b566df)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-npe-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


omit 8315d894b5 Fix the NPE in minimizeDataMovement instance assignment 
strategy
 new 1841b566df Fix the NPE in minimizeDataMovement instance assignment 
strategy

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (8315d894b5)
\
 N -- N -- N   refs/heads/fix-npe-for-minimizeDataMovement 
(1841b566df)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../instance/InstanceReplicaGroupPartitionSelectorTest.java  | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch fix-npe-for-minimizeDataMovement updated (d69374c10d -> 8315d894b5)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-npe-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


omit d69374c10d Fix the NPE in minimizeDataMovement instance assignment 
strategy
 add 8315d894b5 Fix the NPE in minimizeDataMovement instance assignment 
strategy

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (d69374c10d)
\
 N -- N -- N   refs/heads/fix-npe-for-minimizeDataMovement 
(8315d894b5)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../instance/InstanceReplicaGroupPartitionSelectorTest.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) branch fix-npe-for-minimizeDataMovement updated (d9832a4ac4 -> d69374c10d)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-npe-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


omit d9832a4ac4 Fix the NPE in minimizeDataMovement instance assignment 
strategy
 add d69374c10d Fix the NPE in minimizeDataMovement instance assignment 
strategy

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (d9832a4ac4)
\
 N -- N -- N   refs/heads/fix-npe-for-minimizeDataMovement 
(d69374c10d)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .../assignment/instance/InstanceReplicaGroupPartitionSelectorTest.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

(pinot) 01/01: Fix the NPE in minimizeDataMovement instance assignment strategy

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch fix-npe-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit d9832a4ac443f7cd4fed1f3fa4db55f49507a86a
Author: jlli_LinkedIn 
AuthorDate: Fri Nov 3 15:48:31 2023 -0700

Fix the NPE in minimizeDataMovement instance assignment strategy
---
 .../InstanceReplicaGroupPartitionSelector.java |  11 ++-
 .../InstanceReplicaGroupPartitionSelectorTest.java | 105 +
 2 files changed, 111 insertions(+), 5 deletions(-)

diff --git 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
index b6c62ac12e..de1e681d17 100644
--- 
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
+++ 
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelector.java
@@ -72,11 +72,17 @@ public class InstanceReplicaGroupPartitionSelector extends 
InstancePartitionSele
   Preconditions.checkState(numReplicaGroups > 0, "Number of replica-groups 
must be positive");
   Map> poolToReplicaGroupIdsMap = new TreeMap<>();
   Map replicaGroupIdToPoolMap = new TreeMap<>();
+  Map> poolToCandidateInstancesMap = new TreeMap<>();
   for (int replicaId = 0; replicaId < numReplicaGroups; replicaId++) {
 // Pick one pool for each replica-group based on the table name hash
 int pool = pools.get((tableNameHash + replicaId) % numPools);
 poolToReplicaGroupIdsMap.computeIfAbsent(pool, k -> new 
ArrayList<>()).add(replicaId);
 replicaGroupIdToPoolMap.put(replicaId, pool);
+
+Set candidateInstances =
+poolToCandidateInstancesMap.computeIfAbsent(pool, k -> new 
LinkedHashSet<>());
+List instanceConfigsInPool = 
poolToInstanceConfigsMap.get(pool);
+instanceConfigsInPool.forEach(k -> 
candidateInstances.add(k.getInstanceName()));
   }
   LOGGER.info("Selecting {} replica-groups from pool: {} for table: {}", 
numReplicaGroups, poolToReplicaGroupIdsMap,
   _tableNameWithType);
@@ -132,7 +138,6 @@ public class InstanceReplicaGroupPartitionSelector extends 
InstancePartitionSele
 int existingNumReplicaGroups = 
_existingInstancePartitions.getNumReplicaGroups();
 int numCommonReplicaGroups = Math.min(numReplicaGroups, 
existingNumReplicaGroups);
 
-Map> poolToCandidateInstancesMap = new 
TreeMap<>();
 Map> replicaGroupIdToExistingInstancesMap = new 
TreeMap<>();
 // Step 1: find out the replica groups and their existing instances,
 //   so that these instances can be filtered out and won't be chosen 
for the other replica group.
@@ -142,10 +147,6 @@ public class InstanceReplicaGroupPartitionSelector extends 
InstancePartitionSele
 // Skip the replica group if it's no longer needed.
 continue;
   }
-  Set candidateInstances =
-  poolToCandidateInstancesMap.computeIfAbsent(pool, k -> new 
LinkedHashSet<>());
-  List instanceConfigsInPool = 
poolToInstanceConfigsMap.get(pool);
-  instanceConfigsInPool.forEach(k -> 
candidateInstances.add(k.getInstanceName()));
 
   for (int partitionId = 0; partitionId < existingNumPartitions; 
partitionId++) {
 List existingInstances = 
_existingInstancePartitions.getInstances(partitionId, replicaGroupId);
diff --git 
a/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelectorTest.java
 
b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelectorTest.java
new file mode 100644
index 00..4f982c45e6
--- /dev/null
+++ 
b/pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/instance/InstanceReplicaGroupPartitionSelectorTest.java
@@ -0,0 +1,105 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT W

(pinot) branch fix-npe-for-minimizeDataMovement created (now d9832a4ac4)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-npe-for-minimizeDataMovement
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at d9832a4ac4 Fix the NPE in minimizeDataMovement instance assignment 
strategy

This branch includes the following new commits:

 new d9832a4ac4 Fix the NPE in minimizeDataMovement instance assignment 
strategy

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[pinot] branch fix-protobuf-conflict deleted (was 6e7fccea08)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-protobuf-conflict
in repository https://gitbox.apache.org/repos/asf/pinot.git


 was 6e7fccea08 Resolve dependency conflict in pinot-protobuf module

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[pinot] branch master updated: Resolve dependency conflict in pinot-protobuf module (#11867)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git


The following commit(s) were added to refs/heads/master by this push:
 new be74dc1429 Resolve dependency conflict in pinot-protobuf module 
(#11867)
be74dc1429 is described below

commit be74dc1429e4a72c2d51946167c281f039b60165
Author: Jialiang Li 
AuthorDate: Tue Oct 24 17:35:29 2023 -0700

Resolve dependency conflict in pinot-protobuf module (#11867)

Co-authored-by: Jialiang Li 
---
 pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml 
b/pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml
index 7783fb6328..8ba36a7a49 100644
--- a/pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml
+++ b/pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml
@@ -53,6 +53,12 @@
 
   com.google.protobuf
   protobuf-java
+  
+
+  com.google.j2objc
+  j2objc-annotations
+
+  
 
 
   com.github.os72
@@ -132,6 +138,10 @@
   com.google.errorprone
   error_prone_annotations
 
+
+  com.google.j2objc
+  j2objc-annotations
+
   
 
 


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[pinot] branch fix-protobuf-conflict created (now 6e7fccea08)

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a change to branch fix-protobuf-conflict
in repository https://gitbox.apache.org/repos/asf/pinot.git


  at 6e7fccea08 Resolve dependency conflict in pinot-protobuf module

This branch includes the following new commits:

 new 6e7fccea08 Resolve dependency conflict in pinot-protobuf module

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

[pinot] 01/01: Resolve dependency conflict in pinot-protobuf module

[jlli]

This is an automated email from the ASF dual-hosted git repository.

jlli pushed a commit to branch fix-protobuf-conflict
in repository https://gitbox.apache.org/repos/asf/pinot.git

commit 6e7fccea08ab2925df0c987357d340d2ad60c1e6
Author: Jialiang Li 
AuthorDate: Tue Oct 24 14:33:27 2023 -0700

Resolve dependency conflict in pinot-protobuf module
---
 pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml 
b/pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml
index 7783fb6328..8ba36a7a49 100644
--- a/pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml
+++ b/pinot-plugins/pinot-input-format/pinot-protobuf/pom.xml
@@ -53,6 +53,12 @@
 
   com.google.protobuf
   protobuf-java
+  
+
+  com.google.j2objc
+  j2objc-annotations
+
+  
 
 
   com.github.os72
@@ -132,6 +138,10 @@
   com.google.errorprone
   error_prone_annotations
 
+
+  com.google.j2objc
+  j2objc-annotations
+
   
 
 


-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org