Re: [PR] Hash Function for UUID Primary Keys [pinot]
ankitsultana commented on code in PR #12538:
URL: https://github.com/apache/pinot/pull/12538#discussion_r1566736270
##
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java:
##
@@ -44,8 +73,31 @@ public static Object hashPrimaryKey(PrimaryKey primaryKey,
HashFunction hashFunc
return new ByteArray(HashUtils.hashMD5(primaryKey.asBytes()));
case MURMUR3:
return new ByteArray(HashUtils.hashMurmur3(primaryKey.asBytes()));
+ case UUID:
+return new ByteArray(HashUtils.hashUUID(primaryKey.getValues()));
default:
throw new IllegalArgumentException(String.format("Unrecognized hash
function %s", hashFunction));
}
}
+
+ /**
+ * Concatenates the string representation of all values into a single byte
array. Each element is prepended with its
+ * 4-byte length to ensure no collisions can happen. (sacrifice space
efficiency for correctness)
+ */
+ private static byte[] concatenate(Object[] values) {
+byte[][] allValueBytes = new byte[values.length][];
+int totalLen = 0;
+for (int j = 0; j < allValueBytes.length; j++) {
+ allValueBytes[j] = values[j] == null ?
"null".getBytes(StandardCharsets.UTF_8)
Review Comment:
@Jackie-Jiang : any suggestions on the path forward here? I am open to any
of the following options:
* Disambiguate between "null" and null. I can denote a null using a single 0
byte.
* Throw an error if there's a null in keeping with `PrimaryKey`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] [Draft] track the use of partition upsert metadata mgr to reset preloading flag [pinot]
codecov-commenter commented on PR #12938: URL: https://github.com/apache/pinot/pull/12938#issuecomment-2058266890 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12938?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `28.0%` with `18 lines` in your changes are missing coverage. Please review. > Project coverage is 62.18%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`19e7268`)](https://app.codecov.io/gh/apache/pinot/pull/12938?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 310 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/12938?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [...ata/manager/realtime/RealtimeTableDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12938?src=pr=tree=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2Frealtime%2FRealtimeTableDataManager.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12938?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [...cal/upsert/BasePartitionUpsertMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12938?src=pr=tree=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fupsert%2FBasePartitionUpsertMetadataManager.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvQmFzZVBhcnRpdGlvblVwc2VydE1ldGFkYXRhTWFuYWdlci5qYXZh) | 30.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12938?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [...a/manager/realtime/RealtimeSegmentDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12938?src=pr=tree=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2Frealtime%2FRealtimeSegmentDataManager.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | 0.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12938?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [...psert/ConcurrentMapTableUpsertMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12938?src=pr=tree=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fupsert%2FConcurrentMapTableUpsertMetadataManager.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvQ29uY3VycmVudE1hcFRhYmxlVXBzZXJ0TWV0YWRhdGFNYW5hZ2VyLmphdmE=) | 66.66% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12938?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12938 +/- ## + Coverage 61.75% 62.18% +0.43% + Complexity 207 198 -9 Files 2436 2503 +67 Lines133233 136498+3265 Branches 2063621119 +483 + Hits 8227484887+2613 - Misses4491145314 +403 - Partials 6048 6297 +249 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12938/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12938/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | |
Re: [PR] Skip node level throttle during server bootstrap - Enhancement [pinot]
codecov-commenter commented on PR #12930: URL: https://github.com/apache/pinot/pull/12930#issuecomment-2058261465 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12930?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `0%` with `1 lines` in your changes are missing coverage. Please review. > Project coverage is 27.91%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`77bd980`)](https://app.codecov.io/gh/apache/pinot/pull/12930?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 310 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/12930?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [.../pinot/server/starter/helix/BaseServerStarter.java](https://app.codecov.io/gh/apache/pinot/pull/12930?src=pr=tree=pinot-server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fserver%2Fstarter%2Fhelix%2FBaseServerStarter.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12930?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12930 +/- ## = - Coverage 61.75% 27.91% -33.84% + Complexity 207 192 -15 = Files 2436 2502 +66 Lines133233 136388 +3155 Branches 2063621106 +470 = - Hits 8227438070-44204 - Misses4491195417+50506 + Partials 6048 2901 -3147 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `27.91% <0.00%> (-33.72%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `27.91% <0.00%> (-33.84%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `27.91% <0.00%> (-33.84%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `27.91% <0.00%> (-33.84%)` | :arrow_down: | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12930/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `27.91% <0.00%> (+0.18%)` | :arrow_up: | Flags with carried forward
Re: [PR] Bump org.codehaus.mojo:buildnumber-maven-plugin from 1.3 to 3.2.0 [pinot]
codecov-commenter commented on PR #12937: URL: https://github.com/apache/pinot/pull/12937#issuecomment-2058246151 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12937?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`d692271`)](https://app.codecov.io/gh/apache/pinot/pull/12937?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 310 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12937 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2428-8 Lines133233 132835 -398 Branches 2063620568 -68 = - Hits 822740-82274 - Misses44911 132835+87924 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <ø> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <ø> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <ø> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <ø> (-61.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <ø> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12937/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/12937?dropdown=coverage=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service,
Re: [PR] Update superset docker build script [pinot]
xiangfu0 merged PR #12385: URL: https://github.com/apache/pinot/pull/12385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (ec452a49f3 -> 3c45469c39)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
from ec452a49f3 Refine PeerServerSegmentFinder (#12933)
add 3c45469c39 Update superset docker build script (#12385)
No new revisions were added by this update.
Summary of changes:
.../workflows/scripts/docker/.superset_docker_image_build_and_push.sh | 2 +-
..._and_push.sh => .superset_multi_arch_docker_image_build_and_push.sh} | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
copy
.github/workflows/scripts/docker/{.superset_docker_image_build_and_push.sh =>
.superset_multi_arch_docker_image_build_and_push.sh} (98%)
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] [Draft] track the use of partition upsert metadata mgr to reset preloading flag [pinot]
klsince opened a new pull request, #12938: URL: https://github.com/apache/pinot/pull/12938 Try to use refCnt to track threads using partition upsert metadata mgr so that we can be sure when we can reset the isPreloading flag, so that the mgr can load segments via preloading again, e.g. when the table partition is rebalanced back to the old server. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump com.tdunning:t-digest from 3.2 to 3.3 [pinot]
xiangfu0 closed pull request #12882: Bump com.tdunning:t-digest from 3.2 to 3.3 URL: https://github.com/apache/pinot/pull/12882 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump org.codehaus.mojo:buildnumber-maven-plugin from 1.3 to 3.2.0 [pinot]
xiangfu0 opened a new pull request, #12937: URL: https://github.com/apache/pinot/pull/12937 Bumps [org.codehaus.mojo:buildnumber-maven-plugin](https://github.com/mojohaus/buildnumber-maven-plugin) from 1.3 to 3.2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add splitPartWithLimit and splitPartFromEnd UDFs [pinot]
deemoliu commented on code in PR #12437:
URL: https://github.com/apache/pinot/pull/12437#discussion_r1566583241
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -597,6 +596,39 @@ public static String splitPart(String input, String
delimiter, int index) {
}
}
+ /**
+ * @param input the input String to be split into parts.
+ * @param delimiter the specified delimiter to split the input string.
+ * @param index the specified index for the splitted parts to be returned.
+ * @param max the max count of parts that the input string can be splitted
into.
+ * @return splits string on the delimiter with the limit count and returns
String at specified index from the split.
+ */
+ @ScalarFunction
+ public static String splitPart(String input, String delimiter, int index,
int max) {
+String[] splitString = StringUtils.splitByWholeSeparator(input, delimiter,
max);
+if (index < splitString.length) {
+ return splitString[index];
+} else {
+ return "null";
+}
+ }
+
+ /**
+ * @param input the input String to be split into parts.
+ * @param delimiter the specified delimiter to split the input string.
+ * @param index the specified index for the splitted parts to be returned.
+ * @return splits string on the delimiter with the limit count and returns
String at specified index from the split.
+ */
+ @ScalarFunction
+ public static String splitPartFromEnd(String input, String delimiter, int
index) {
Review Comment:
@Jackie-Jiang, after i adopt to your suggestion, this is a backward
incompatible behavior since previously the negative value index will error out
(out of range). now these values will indicates the index from end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add splitPartWithLimit and splitPartFromEnd UDFs [pinot]
deemoliu commented on code in PR #12437:
URL: https://github.com/apache/pinot/pull/12437#discussion_r1566582209
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -597,6 +596,39 @@ public static String splitPart(String input, String
delimiter, int index) {
}
}
+ /**
+ * @param input the input String to be split into parts.
+ * @param delimiter the specified delimiter to split the input string.
+ * @param index the specified index for the splitted parts to be returned.
+ * @param max the max count of parts that the input string can be splitted
into.
+ * @return splits string on the delimiter with the limit count and returns
String at specified index from the split.
+ */
+ @ScalarFunction
+ public static String splitPart(String input, String delimiter, int index,
int max) {
Review Comment:
updated, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add splitPartWithLimit and splitPartFromEnd UDFs [pinot]
deemoliu commented on code in PR #12437:
URL: https://github.com/apache/pinot/pull/12437#discussion_r1566580709
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -597,6 +596,39 @@ public static String splitPart(String input, String
delimiter, int index) {
}
}
+ /**
+ * @param input the input String to be split into parts.
+ * @param delimiter the specified delimiter to split the input string.
+ * @param index the specified index for the splitted parts to be returned.
+ * @param max the max count of parts that the input string can be splitted
into.
+ * @return splits string on the delimiter with the limit count and returns
String at specified index from the split.
+ */
+ @ScalarFunction
+ public static String splitPart(String input, String delimiter, int index,
int max) {
+String[] splitString = StringUtils.splitByWholeSeparator(input, delimiter,
max);
+if (index < splitString.length) {
+ return splitString[index];
+} else {
+ return "null";
+}
+ }
+
+ /**
+ * @param input the input String to be split into parts.
+ * @param delimiter the specified delimiter to split the input string.
+ * @param index the specified index for the splitted parts to be returned.
+ * @return splits string on the delimiter with the limit count and returns
String at specified index from the split.
Review Comment:
updated, thanks
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -597,6 +596,39 @@ public static String splitPart(String input, String
delimiter, int index) {
}
}
+ /**
+ * @param input the input String to be split into parts.
+ * @param delimiter the specified delimiter to split the input string.
+ * @param index the specified index for the splitted parts to be returned.
+ * @param max the max count of parts that the input string can be splitted
into.
+ * @return splits string on the delimiter with the limit count and returns
String at specified index from the split.
Review Comment:
updated, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Refine SegmentFetcherFactory [pinot]
codecov-commenter commented on PR #12936: URL: https://github.com/apache/pinot/pull/12936#issuecomment-2058042624 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12936?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `0%` with `33 lines` in your changes are missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`b7d20ca`)](https://app.codecov.io/gh/apache/pinot/pull/12936?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 309 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/12936?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [...ot/common/utils/fetcher/SegmentFetcherFactory.java](https://app.codecov.io/gh/apache/pinot/pull/12936?src=pr=tree=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Futils%2Ffetcher%2FSegmentFetcherFactory.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZmV0Y2hlci9TZWdtZW50RmV0Y2hlckZhY3RvcnkuamF2YQ==) | 0.00% | [23 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12936?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [.../pinot/core/data/manager/BaseTableDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12936?src=pr=tree=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2FBaseTableDataManager.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12936?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12936 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2428-8 Lines133233 132819 -414 Branches 2063620567 -69 = - Hits 822740-82274 - Misses44911 132819+87908 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-61.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-27.73%)` | :arrow_down: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12936/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%>
Re: [PR] Add splitPartWithLimit and splitPartFromEnd UDFs [pinot]
deemoliu commented on code in PR #12437:
URL: https://github.com/apache/pinot/pull/12437#discussion_r1566580853
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -597,6 +596,39 @@ public static String splitPart(String input, String
delimiter, int index) {
}
}
+ /**
+ * @param input the input String to be split into parts.
+ * @param delimiter the specified delimiter to split the input string.
+ * @param index the specified index for the splitted parts to be returned.
+ * @param max the max count of parts that the input string can be splitted
into.
+ * @return splits string on the delimiter with the limit count and returns
String at specified index from the split.
+ */
+ @ScalarFunction
+ public static String splitPart(String input, String delimiter, int index,
int max) {
+String[] splitString = StringUtils.splitByWholeSeparator(input, delimiter,
max);
+if (index < splitString.length) {
+ return splitString[index];
+} else {
+ return "null";
+}
+ }
+
+ /**
+ * @param input the input String to be split into parts.
+ * @param delimiter the specified delimiter to split the input string.
+ * @param index the specified index for the splitted parts to be returned.
+ * @return splits string on the delimiter with the limit count and returns
String at specified index from the split.
+ */
+ @ScalarFunction
+ public static String splitPartFromEnd(String input, String delimiter, int
index) {
Review Comment:
updated, thanks @Jackie-Jiang can you review again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1566574337
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/MultiStageQueryStats.java:
##
@@ -0,0 +1,668 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.runtime.plan;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.google.common.base.Preconditions;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.avro.util.ByteBufferInputStream;
+import org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.query.runtime.operator.BaseMailboxReceiveOperator;
+import
org.apache.pinot.query.runtime.operator.LeafStageTransferableBlockOperator;
+import org.apache.pinot.query.runtime.operator.LiteralValueOperator;
+import org.apache.pinot.query.runtime.operator.MailboxSendOperator;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The stats of a given query.
+ *
+ * For the same query, multiple instances of this class may exist. Each of
them will have a partial view of the stats.
+ * Specifically, while the query is being executed, each operator will return
its own partial view of the stats when
+ * EOS block is sent.
+ *
+ * Simple operations with a single upstream, like filters or transforms, would
just add their own information to the
+ * stats. More complex operations, like joins or receiving mailboxes, will
merge the stats from all their upstreams and
+ * add their own stats.
+ *
+ * The complete stats for the query are obtained in the execution root
(usually the broker) by merging the partial
+ * views.
+ *
+ * In order to reduce allocation, this class is mutable. Some operators may
create their own stats, but most of them
+ * will receive a stats object from the upstream operator and modify it by
adding their own stats and sometimes merging
+ * them with other upstream stats.
+ */
+public class MultiStageQueryStats {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(MultiStageQueryStats.class);
+ private final int _currentStageId;
+ private final StageStats.Open _currentStats;
+ /**
+ * Known stats for stages whose id is higher than the current one.
+ *
+ * A stage may not know all the stats whose id is higher than the current
one, so this list may contain null values.
+ * It may also grow in size when different merge methods are called.
+ *
+ * For example the stats of the left hand side of a join may know stats of
stages 3 and 4 and the right side may know
+ * stats of stages 5. When merging the stats of the join, the stats of
stages 5 will be added to this list.
+ *
+ * @see #mergeUpstream(List)
+ * @see #mergeUpstream(MultiStageQueryStats)
+ * @see #mergeInOrder(MultiStageQueryStats, MultiStageOperator.Type, StatMap)
+ */
+ private final ArrayList _closedStats;
+ private static final MultiStageOperator.Type[] ALL_TYPES =
MultiStageOperator.Type.values();
+
+ private MultiStageQueryStats(int stageId) {
+_currentStageId = stageId;
+_currentStats = new StageStats.Open();
+_closedStats = new ArrayList<>();
+ }
+
+ private static MultiStageQueryStats create(int stageId,
MultiStageOperator.Type type, @Nullable StatMap opStats) {
+MultiStageQueryStats multiStageQueryStats = new
MultiStageQueryStats(stageId);
+
Re: [PR] Multi stage stats [pinot]
Jackie-Jiang commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1566570540
##
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER =
LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
Review Comment:
We use version to maintain backward compatible. When introducing a new
version, we first add the support for both versions, then after all nodes are
upgraded, use a flag to switch the versions. After it is stable, remove the
flag and old version support.
Since we are not following the above process, version is not really checked
when deserializing. I don't have strong opinion on whether to bump it. Just
pointing it out that it is not necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add Prefix, Suffix and Ngram UDFs [pinot]
Jackie-Jiang commented on code in PR #12392:
URL: https://github.com/apache/pinot/pull/12392#discussion_r1566565835
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -581,6 +586,107 @@ public static String[] split(String input, String
delimiter, int limit) {
return StringUtils.splitByWholeSeparator(input, delimiter, limit);
}
+ /**
+ * @param input an input string for prefix strings generations.
+ * @param maxlength the max length of the prefix strings for the string.
+ * @return generate an array of prefix strings of the string that are
shorter than the specified length.
+ */
+ @ScalarFunction
+ public static String[] prefixes(String input, int maxlength) {
+ObjectList prefixList = new ObjectArrayList<>();
Review Comment:
`ObjectArrayList` is not really buying us anything here.
Given we know the number of prefixes upfront, we can directly allocate the
array
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -18,6 +18,10 @@
*/
package org.apache.pinot.common.function.scalar;
+import it.unimi.dsi.fastutil.objects.ObjectArrayList;
Review Comment:
No need to use fastutil here. We can use the java default ones
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -570,6 +572,107 @@ public static String[] split(String input, String
delimiter, int limit) {
return StringUtils.splitByWholeSeparator(input, delimiter, limit);
}
+ /**
+ * @param input an input string for prefix strings generations.
+ * @param maxlength the max length of the prefix strings for the string.
+ * @return generate an array of prefix strings of the string that are
shorter than the specified length.
+ */
+ @ScalarFunction
+ public static String[] uniquePrefixes(String input, int maxlength) {
+ObjectSet prefixSet = new ObjectLinkedOpenHashSet<>();
+for (int prefixLength = 1; prefixLength <= maxlength && prefixLength <=
input.length(); prefixLength++) {
+ prefixSet.add(input.substring(0, prefixLength));
+}
+return prefixSet.toArray(new String[0]);
+ }
+
+ /**
+ * @param input an input string for prefix strings generations.
+ * @param maxlength the max length of the prefix strings for the string.
+ * @param prefix the prefix to be prepended to prefix strings generated.
e.g. '^' for regex matching
+ * @return generate an array of prefix matchers of the string that are
shorter than the specified length.
+ */
+ @ScalarFunction
+ public static String[] uniquePrefixesWithPrefix(String input, int maxlength,
String prefix) {
+if (prefix == null) {
Review Comment:
This is not addressed ^^
Take a look at `ScalarFunction.class`. You need to annotate it as
`nullableParameters`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Refine SegmentFetcherFactory [pinot]
Jackie-Jiang opened a new pull request, #12936: URL: https://github.com/apache/pinot/pull/12936 Separated from #12886 for easier review - Make `SegmentFetcherFactory` a util class - Integrate the new `API` with URI supplier from `SegmentFetcher` into `SegmentFetcherFactory` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Refine PeerServerSegmentFinder (#12933)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new ec452a49f3 Refine PeerServerSegmentFinder (#12933)
ec452a49f3 is described below
commit ec452a49f3c885308613bc45dfa44b48a16076ba
Author: Xiaotian (Jackie) Jiang <1751+jackie-ji...@users.noreply.github.com>
AuthorDate: Mon Apr 15 17:02:18 2024 -0700
Refine PeerServerSegmentFinder (#12933)
---
.../common/utils/fetcher/BaseSegmentFetcher.java | 9 +-
.../common/utils/fetcher/HttpSegmentFetcher.java | 28 ++--
.../pinot/core/util/PeerServerSegmentFinder.java | 101 ++
.../utils/fetcher/HttpSegmentFetcherTest.java | 152 +++--
.../realtime/PinotLLCRealtimeSegmentManager.java | 3 +-
.../PinotLLCRealtimeSegmentManagerTest.java| 91 +---
.../core/data/manager/BaseTableDataManager.java| 9 +-
.../manager/realtime/RealtimeTableDataManager.java | 28 ++--
.../data/manager/BaseTableDataManagerTest.java | 4 +-
.../core/util/PeerServerSegmentFinderTest.java | 128 -
.../utils/retry/ExponentialBackoffRetryPolicy.java | 6 +-
11 files changed, 220 insertions(+), 339 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java
index d33c7ead43..5fb82388f2 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java
@@ -42,13 +42,13 @@ public abstract class BaseSegmentFetcher implements
SegmentFetcher {
public static final String RETRY_DELAY_SCALE_FACTOR_CONFIG_KEY =
"retry.delay.scale.factor";
public static final int DEFAULT_RETRY_COUNT = 3;
public static final int DEFAULT_RETRY_WAIT_MS = 100;
- public static final int DEFAULT_RETRY_DELAY_SCALE_FACTOR = 5;
+ public static final double DEFAULT_RETRY_DELAY_SCALE_FACTOR = 5;
protected final Logger _logger =
LoggerFactory.getLogger(getClass().getSimpleName());
protected int _retryCount;
protected int _retryWaitMs;
- protected int _retryDelayScaleFactor;
+ protected double _retryDelayScaleFactor;
protected AuthProvider _authProvider;
@Override
@@ -58,9 +58,8 @@ public abstract class BaseSegmentFetcher implements
SegmentFetcher {
_retryDelayScaleFactor =
config.getProperty(RETRY_DELAY_SCALE_FACTOR_CONFIG_KEY,
DEFAULT_RETRY_DELAY_SCALE_FACTOR);
_authProvider = AuthProviderUtils.extractAuthProvider(config,
CommonConstants.KEY_OF_AUTH);
doInit(config);
-_logger
-.info("Initialized with retryCount: {}, retryWaitMs: {},
retryDelayScaleFactor: {}", _retryCount, _retryWaitMs,
-_retryDelayScaleFactor);
+_logger.info("Initialized with retryCount: {}, retryWaitMs: {},
retryDelayScaleFactor: {}", _retryCount,
+_retryWaitMs, _retryDelayScaleFactor);
}
/**
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpSegmentFetcher.java
b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpSegmentFetcher.java
index 170327dc5b..6872ac7714 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpSegmentFetcher.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/HttpSegmentFetcher.java
@@ -44,23 +44,16 @@ import org.apache.pinot.spi.utils.retry.RetryPolicies;
public class HttpSegmentFetcher extends BaseSegmentFetcher {
protected FileUploadDownloadClient _httpClient;
- @Override
- protected void doInit(PinotConfiguration config) {
-_httpClient = new
FileUploadDownloadClient(HttpClientConfig.newBuilder(config).build());
- }
-
- public HttpSegmentFetcher() {
- }
-
@VisibleForTesting
- protected HttpSegmentFetcher(FileUploadDownloadClient httpClient,
PinotConfiguration config) {
+ void setHttpClient(FileUploadDownloadClient httpClient) {
_httpClient = httpClient;
-_retryCount = config.getProperty(RETRY_COUNT_CONFIG_KEY,
DEFAULT_RETRY_COUNT);
-_retryWaitMs = config.getProperty(RETRY_WAIT_MS_CONFIG_KEY,
DEFAULT_RETRY_WAIT_MS);
-_retryDelayScaleFactor =
config.getProperty(RETRY_DELAY_SCALE_FACTOR_CONFIG_KEY,
DEFAULT_RETRY_DELAY_SCALE_FACTOR);
-_logger
-.info("Initialized with retryCount: {}, retryWaitMs: {},
retryDelayScaleFactor: {}", _retryCount, _retryWaitMs,
-_retryDelayScaleFactor);
+ }
+
+ @Override
+ protected void doInit(PinotConfiguration config) {
+if (_httpClient == null) {
+ _httpClient = new
FileUploadDownloadClient(HttpClientConfig.newBuilder(config).build());
+}
}
@Override
@@ -87,9 +80,8 @@ public class HttpSegmentFetcher extends BaseSegmentFetcher {
Re: [PR] Refine PeerServerSegmentFinder [pinot]
Jackie-Jiang merged PR #12933: URL: https://github.com/apache/pinot/pull/12933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (edf9d53597 -> c08ba2ccd5)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from edf9d53597 Specify version for commons-validator (#12935) add c08ba2ccd5 Bump org.apache.maven.scm:maven-scm-provider-gitexe from 2.0.1 to 2.1.0 (#12925) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.apache.maven.scm-maven-scm-provider-gitexe-2.1.0 deleted (was 068abcfc4d)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.apache.maven.scm-maven-scm-provider-gitexe-2.1.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 068abcfc4d Bump org.apache.maven.scm:maven-scm-provider-gitexe from 2.0.1 to 2.1.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump org.apache.maven.scm:maven-scm-provider-gitexe from 2.0.1 to 2.1.0 [pinot]
Jackie-Jiang merged PR #12925: URL: https://github.com/apache/pinot/pull/12925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Specify version for commons-validator (#12935)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new edf9d53597 Specify version for commons-validator (#12935)
edf9d53597 is described below
commit edf9d535975426a106991e3f61cce29079b47539
Author: Xiaotian (Jackie) Jiang <1751+jackie-ji...@users.noreply.github.com>
AuthorDate: Mon Apr 15 16:58:11 2024 -0700
Specify version for commons-validator (#12935)
---
pom.xml | 6 ++
1 file changed, 6 insertions(+)
diff --git a/pom.xml b/pom.xml
index 0c51fca074..6636cdaa89 100644
--- a/pom.xml
+++ b/pom.xml
@@ -187,6 +187,7 @@
1.16.1
1.6.0
3.10.0
+1.8.0
2.6
3.2.2
@@ -728,6 +729,11 @@
commons-net
${commons-net.version}
+
+commons-validator
+commons-validator
+${commons-validator.version}
+
commons-lang
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Specify version for commons-validator [pinot]
Jackie-Jiang merged PR #12935: URL: https://github.com/apache/pinot/pull/12935 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Specify version for commons-validator [pinot]
Jackie-Jiang opened a new pull request, #12935: URL: https://github.com/apache/pinot/pull/12935 Solves the dependency conflict from confluent libraries -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (684cece35e -> 013435a6a1)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 684cece35e Bump org.apache.maven.plugins:maven-jar-plugin from 3.3.0 to 3.4.0 (#12927) add 013435a6a1 Bump aws.sdk.version from 2.25.30 to 2.25.31 (#12926) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/aws.sdk.version-2.25.31 deleted (was 5863c220e8)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/aws.sdk.version-2.25.31 in repository https://gitbox.apache.org/repos/asf/pinot.git was 5863c220e8 Bump aws.sdk.version from 2.25.30 to 2.25.31 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Pulsar test instability [pinot]
Jackie-Jiang commented on issue #8537: URL: https://github.com/apache/pinot/issues/8537#issuecomment-2057959769 Still flaky.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Bump org.apache.maven.plugins:maven-jar-plugin from 3.3.0 to 3.4.0 (#12927)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 684cece35e Bump org.apache.maven.plugins:maven-jar-plugin from 3.3.0 to 3.4.0 (#12927) 684cece35e is described below commit 684cece35eb790b971f87952c2f2e260b0539c2e Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Apr 15 16:07:59 2024 -0700 Bump org.apache.maven.plugins:maven-jar-plugin from 3.3.0 to 3.4.0 (#12927) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 22daf3bb30..7f82ab 100644 --- a/pom.xml +++ b/pom.xml @@ -130,7 +130,7 @@ org.apache.pinot.shaded -3.3.0 +3.4.0 1.11.3 1.13.1 - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump slf4j.version from 2.0.12 to 2.0.13 [pinot]
Jackie-Jiang merged PR #12928: URL: https://github.com/apache/pinot/pull/12928 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump aws.sdk.version from 2.25.30 to 2.25.31 [pinot]
Jackie-Jiang merged PR #12926: URL: https://github.com/apache/pinot/pull/12926 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[I] Pulsar test instability [pinot]
walterddr opened a new issue, #8537: URL: https://github.com/apache/pinot/issues/8537 Error is occurring to Pulsar test case randomly on multiple of my PRs ``` PulsarConsumerTest.testPartitionLevelConsumerBatchMessages:246 expected [500] but found [491] ``` CI Logs. Click to expand ``` Error: Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.324 s <<< FAILURE! - in org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest Error: org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest.testPartitionLevelConsumer Time elapsed: 0.845 s Error: org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest.testPartitionLevelConsumerBatchMessages Time elapsed: 0.48 s <<< FAILURE! java.lang.AssertionError: expected [500] but found [491] at org.testng.Assert.fail(Assert.java:93) at org.testng.Assert.failNotEquals(Assert.java:512) at org.testng.Assert.assertEqualsImpl(Assert.java:134) at org.testng.Assert.assertEquals(Assert.java:115) at org.testng.Assert.assertEquals(Assert.java:388) at org.testng.Assert.assertEquals(Assert.java:398) at org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest.testPartitionLevelConsumerBatchMessages(PulsarConsumerTest.java:246) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108) at org.testng.internal.Invoker.invokeMethod(Invoker.java:661) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) at org.testng.TestRunner.privateRun(TestRunner.java:744) at org.testng.TestRunner.run(TestRunner.java:602) at org.testng.SuiteRunner.runTest(SuiteRunner.java:380) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340) at org.testng.SuiteRunner.run(SuiteRunner.java:289) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301) at org.testng.TestNG.runSuitesLocally(TestNG.java:1226) at org.testng.TestNG.runSuites(TestNG.java:1144) at org.testng.TestNG.run(TestNG.java:1115) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:136) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:145) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162) at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548) [INFO] [INFO] Results: [INFO] Error: Failures: Error:PulsarConsumerTest.testPartitionLevelConsumerBatchMessages:246 expected [500] but found [491] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.apache.maven.plugins-maven-jar-plugin-3.4.0 deleted (was 3130039947)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/org.apache.maven.plugins-maven-jar-plugin-3.4.0 in repository https://gitbox.apache.org/repos/asf/pinot.git was 3130039947 Bump org.apache.maven.plugins:maven-jar-plugin from 3.3.0 to 3.4.0 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump org.apache.maven.plugins:maven-jar-plugin from 3.3.0 to 3.4.0 [pinot]
Jackie-Jiang merged PR #12927: URL: https://github.com/apache/pinot/pull/12927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/slf4j.version-2.0.13 deleted (was dfcda2cd41)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/slf4j.version-2.0.13 in repository https://gitbox.apache.org/repos/asf/pinot.git was dfcda2cd41 Bump slf4j.version from 2.0.12 to 2.0.13 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Bump slf4j.version from 2.0.12 to 2.0.13 (#12928)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 2459cfce75 Bump slf4j.version from 2.0.12 to 2.0.13 (#12928) 2459cfce75 is described below commit 2459cfce752aafd0ab6ff3d2f1eb98ac57b7efaf Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Apr 15 16:07:44 2024 -0700 Bump slf4j.version from 2.0.12 to 2.0.13 (#12928) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 7f1819250b..22daf3bb30 100644 --- a/pom.xml +++ b/pom.xml @@ -159,7 +159,7 @@ 1.8.0 0.20.0 2.23.1 -2.0.12 +2.0.13 4.1.109.Final 1.0.4 1.19.0 - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch clean-up-task-manager-for-tables deleted (was 7068a246e9)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch clean-up-task-manager-for-tables in repository https://gitbox.apache.org/repos/asf/pinot.git was 7068a246e9 Do task related cleanup for non-controller leader This change permanently discards the following revisions: discard 7068a246e9 Do task related cleanup for non-controller leader - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot-site) branch new-site-dev updated: ReactPlayer, use Head from nextjs, move it above (#115)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch new-site-dev
in repository https://gitbox.apache.org/repos/asf/pinot-site.git
The following commit(s) were added to refs/heads/new-site-dev by this push:
new 17939a33 ReactPlayer, use Head from nextjs, move it above (#115)
17939a33 is described below
commit 17939a33e846f98303fc426b4a9ff4a82fbbad68
Author: Gio <153032991+gio-start...@users.noreply.github.com>
AuthorDate: Tue Apr 16 00:52:31 2024 +0200
ReactPlayer, use Head from nextjs, move it above (#115)
* Try with ReactPlayer
* httpEquiv -> http-equiv
* http-equiv; use Head from nextjs
---
app/layout.tsx | 13 +++--
app/toberemoved/page.tsx | 30 ++
components/TextVideoSplitSection.tsx | 26 +-
next.config.js | 16
4 files changed, 70 insertions(+), 15 deletions(-)
diff --git a/app/layout.tsx b/app/layout.tsx
index 72915395..03144b83 100644
--- a/app/layout.tsx
+++ b/app/layout.tsx
@@ -10,6 +10,7 @@ import Footer from '@/components/Footer';
import siteMetadata from '@/data/siteMetadata';
import { ThemeProviders } from './theme-providers';
import { Metadata } from 'next';
+import Head from 'next/head';
const work_sans = Work_Sans({
subsets: ['latin'],
@@ -64,7 +65,11 @@ export default function RootLayout({ children }: { children:
React.ReactNode })
className={`${work_sans.variable} scroll-smooth`}
suppressHydrationWarning
>
-
+
+https://www.youtube.com https://youtube.com;;
+/>
-
-
+
diff --git a/app/toberemoved/page.tsx b/app/toberemoved/page.tsx
new file mode 100644
index ..2017d1fb
--- /dev/null
+++ b/app/toberemoved/page.tsx
@@ -0,0 +1,30 @@
+'use client';
+import React, { useEffect, useState } from 'react';
+import ReactPlayer from 'react-player';
+
+const ToBeRemoved = () => {
+const [isSSR, setIsSSR] = useState(true);
+const imagePlaceholder = '/static/images/video_thumbnail.png';
+
+useEffect(() => {
+setIsSSR(false);
+}, []);
+return isSSR ? null : (
+https://www.youtube.com/watch?v=_lqdfq2c9cQ'}
+light={true}
+// width={'720px'}
+// height={'480px'}
+// style={{ border: '1px solid white' }}
+playing={true}
+volume={1}
+pip={true}
+controls={true}
+loop={false}
+stopOnUnmount={true}
+className="h-[197px] w-full"
+/>
+);
+};
+
+export default ToBeRemoved;
diff --git a/components/TextVideoSplitSection.tsx
b/components/TextVideoSplitSection.tsx
index 524ff081..19a663fb 100644
--- a/components/TextVideoSplitSection.tsx
+++ b/components/TextVideoSplitSection.tsx
@@ -1,10 +1,12 @@
'use client';
-import React, { useState } from 'react';
+import React, { useEffect, useState } from 'react';
import { Button } from './ui/button';
import { ArrowRight } from 'lucide-react';
import Link from 'next/link';
import siteMetadata from '@/data/siteMetadata';
+// import ReactPlayer from 'react-player';
+import ReactPlayer from 'react-player/youtube';
interface TextVideoSplitSectionProps {
videoUrl: string;
@@ -13,8 +15,13 @@ interface TextVideoSplitSectionProps {
const TextVideoSplitSection: React.FC = ({
videoUrl, title }) => {
const [iframeLoaded, setIframeLoaded] = useState(false);
+const [isSSR, setIsSSR] = useState(true);
const imagePlaceholder = '/static/images/video_thumbnail.png';
+useEffect(() => {
+setIsSSR(false);
+}, []);
+
const handlePlaceholderClick = () => {
setIframeLoaded(true);
window.open(videoUrl, '_blank');
@@ -60,6 +67,23 @@ const TextVideoSplitSection:
React.FC = ({ videoUrl,
+{/* {isSSR ? null : (
+
+)} */}
+{/* https://www.youtube.com/watch?v=LXb3EKWsInQ;> */}
{
}
]
},
-async headers() {
-return [
-{
-source: '/(.*)',
-headers: securityHeaders
-}
-];
-},
+// async headers() {
+// return [
+// {
+// source: '/(.*)',
+// headers: securityHeaders
+// }
+// ];
+// },
webpack: (config, options) => {
config.module.rules.push({
test: /\.svg$/,
Re: [PR] ReactPlayer, use Head from nextjs, move it above [pinot-site]
xiangfu0 merged PR #115: URL: https://github.com/apache/pinot-site/pull/115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) 01/01: Do task related cleanup for non-controller leader
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch clean-up-task-manager-for-tables
in repository https://gitbox.apache.org/repos/asf/pinot.git
commit 7068a246e9851329a7580cbba4479b8599aa64a2
Author: jlli_LinkedIn
AuthorDate: Mon Apr 15 15:39:13 2024 -0700
Do task related cleanup for non-controller leader
---
.../controller/helix/core/minion/PinotTaskManager.java | 16 +++-
.../helix/core/minion/generator/PinotTaskGenerator.java | 7 +++
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java
index 2cdbf8c1df..a96e550738 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java
@@ -693,11 +693,25 @@ public class PinotTaskManager extends
ControllerPeriodicTask {
scheduleTasks(tableNamesWithType, true);
}
+ @Override
+ protected void nonLeaderCleanup(List tableNamesWithType) {
+LOGGER.info("Cleaning up all task generators for tables: " +
tableNamesWithType);
+for (String taskType : _taskGeneratorRegistry.getAllTaskTypes()) {
+ PinotTaskGenerator pinotTaskGenerator =
_taskGeneratorRegistry.getTaskGenerator(taskType);
+ if (pinotTaskGenerator != null) {
+pinotTaskGenerator.nonLeaderCleanUp(tableNamesWithType);
+ }
+}
+ }
+
@Override
public void cleanUpTask() {
LOGGER.info("Cleaning up all task generators");
for (String taskType : _taskGeneratorRegistry.getAllTaskTypes()) {
- _taskGeneratorRegistry.getTaskGenerator(taskType).nonLeaderCleanUp();
+ PinotTaskGenerator pinotTaskGenerator =
_taskGeneratorRegistry.getTaskGenerator(taskType);
+ if (pinotTaskGenerator != null) {
+pinotTaskGenerator.nonLeaderCleanUp();
+ }
}
}
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/PinotTaskGenerator.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/PinotTaskGenerator.java
index ad42135886..fe98e971f8 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/PinotTaskGenerator.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/PinotTaskGenerator.java
@@ -86,6 +86,13 @@ public interface PinotTaskGenerator {
default void nonLeaderCleanUp() {
}
+ /**
+ * Given the list of table names, performs necessary cleanups (e.g. remove
metrics) when the controller leadership
+ * changes.
+ */
+ default void nonLeaderCleanUp(List tableNameWithType) {
+ }
+
/**
* Gets the minionInstanceTag for the tableConfig
*/
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch clean-up-task-manager-for-tables created (now 7068a246e9)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch clean-up-task-manager-for-tables in repository https://gitbox.apache.org/repos/asf/pinot.git at 7068a246e9 Do task related cleanup for non-controller leader This branch includes the following new commits: new 7068a246e9 Do task related cleanup for non-controller leader The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Move package org.apache.calcite to org.apache.pinot.calcite [pinot]
Jackie-Jiang commented on PR #12837: URL: https://github.com/apache/pinot/pull/12837#issuecomment-2057916250 This change is introducing backward incompatibility because we rely on class path during query plan ser/de, and `PinotRelExchangeType` is part of the ser/de. We should fix the ser/de so that it doesn't serialize the class path for both efficiency and forward compatibility -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add Prefix, Suffix and Ngram UDFs [pinot]
deemoliu commented on code in PR #12392:
URL: https://github.com/apache/pinot/pull/12392#discussion_r1566393961
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -570,6 +572,107 @@ public static String[] split(String input, String
delimiter, int limit) {
return StringUtils.splitByWholeSeparator(input, delimiter, limit);
}
+ /**
+ * @param input an input string for prefix strings generations.
+ * @param maxlength the max length of the prefix strings for the string.
+ * @return generate an array of prefix strings of the string that are
shorter than the specified length.
+ */
+ @ScalarFunction
Review Comment:
sg. i think the reason of unique_prefixes is to reserve `prefixes` for other
purpose or implementations. if no objection, let me use `prefixes()` then.
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -570,6 +572,107 @@ public static String[] split(String input, String
delimiter, int limit) {
return StringUtils.splitByWholeSeparator(input, delimiter, limit);
}
+ /**
+ * @param input an input string for prefix strings generations.
+ * @param maxlength the max length of the prefix strings for the string.
+ * @return generate an array of prefix strings of the string that are
shorter than the specified length.
+ */
+ @ScalarFunction
+ public static String[] uniquePrefixes(String input, int maxlength) {
+ObjectSet prefixSet = new ObjectLinkedOpenHashSet<>();
+for (int prefixLength = 1; prefixLength <= maxlength && prefixLength <=
input.length(); prefixLength++) {
+ prefixSet.add(input.substring(0, prefixLength));
+}
+return prefixSet.toArray(new String[0]);
+ }
+
+ /**
+ * @param input an input string for prefix strings generations.
+ * @param maxlength the max length of the prefix strings for the string.
+ * @param prefix the prefix to be prepended to prefix strings generated.
e.g. '^' for regex matching
+ * @return generate an array of prefix matchers of the string that are
shorter than the specified length.
+ */
+ @ScalarFunction
+ public static String[] uniquePrefixesWithPrefix(String input, int maxlength,
String prefix) {
+if (prefix == null) {
Review Comment:
updated, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Add Prefix, Suffix and Ngram UDFs [pinot]
deemoliu commented on code in PR #12392:
URL: https://github.com/apache/pinot/pull/12392#discussion_r1566496110
##
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/StringFunctions.java:
##
@@ -570,6 +572,107 @@ public static String[] split(String input, String
delimiter, int limit) {
return StringUtils.splitByWholeSeparator(input, delimiter, limit);
}
+ /**
+ * @param input an input string for prefix strings generations.
+ * @param maxlength the max length of the prefix strings for the string.
+ * @return generate an array of prefix strings of the string that are
shorter than the specified length.
+ */
+ @ScalarFunction
+ public static String[] uniquePrefixes(String input, int maxlength) {
+ObjectSet prefixSet = new ObjectLinkedOpenHashSet<>();
Review Comment:
updated, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Refine PeerServerSegmentFinder [pinot]
codecov-commenter commented on PR #12933: URL: https://github.com/apache/pinot/pull/12933#issuecomment-2057865306 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12933?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `0%` with `47 lines` in your changes are missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`246d07f`)](https://app.codecov.io/gh/apache/pinot/pull/12933?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 303 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/12933?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [...pache/pinot/core/util/PeerServerSegmentFinder.java](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Futil%2FPeerServerSegmentFinder.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb3JlL3V0aWwvUGVlclNlcnZlclNlZ21lbnRGaW5kZXIuamF2YQ==) | 0.00% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [...ata/manager/realtime/RealtimeTableDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2Frealtime%2FRealtimeTableDataManager.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [...pinot/common/utils/fetcher/HttpSegmentFetcher.java](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Futils%2Ffetcher%2FHttpSegmentFetcher.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZmV0Y2hlci9IdHRwU2VnbWVudEZldGNoZXIuamF2YQ==) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [.../pinot/core/data/manager/BaseTableDataManager.java](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Fdata%2Fmanager%2FBaseTableDataManager.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [...spi/utils/retry/ExponentialBackoffRetryPolicy.java](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Futils%2Fretry%2FExponentialBackoffRetryPolicy.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvcmV0cnkvRXhwb25lbnRpYWxCYWNrb2ZmUmV0cnlQb2xpY3kuamF2YQ==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [...pinot/common/utils/fetcher/BaseSegmentFetcher.java](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Futils%2Ffetcher%2FBaseSegmentFetcher.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZmV0Y2hlci9CYXNlU2VnbWVudEZldGNoZXIuamF2YQ==) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12933?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | |
Re: [PR] Using local copy of segment instead of downloading from remote [pinot]
klsince merged PR #12863: URL: https://github.com/apache/pinot/pull/12863 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (c823430f11 -> af8fd4089c)
This is an automated email from the ASF dual-hosted git repository. xbli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from c823430f11 Bump io.netty:netty-bom from 4.1.108.Final to 4.1.109.Final (#12929) add af8fd4089c Using local copy of segment instead of downloading from remote (#12863) No new revisions were added by this update. Summary of changes: .../BaseMultipleSegmentsConversionExecutor.java| 26 ++ 1 file changed, 12 insertions(+), 14 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/slf4j.version-2.0.13 updated (6c539cc661 -> dfcda2cd41)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/slf4j.version-2.0.13 in repository https://gitbox.apache.org/repos/asf/pinot.git discard 6c539cc661 Bump slf4j.version from 2.0.12 to 2.0.13 add 2c6a84b8ec Improved segment build time for Lucene text index realtime to offline conversion (#12744) add c823430f11 Bump io.netty:netty-bom from 4.1.108.Final to 4.1.109.Final (#12929) add dfcda2cd41 Bump slf4j.version from 2.0.12 to 2.0.13 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (6c539cc661) \ N -- N -- N refs/heads/dependabot/maven/slf4j.version-2.0.13 (dfcda2cd41) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../indexsegment/mutable/MutableSegmentImpl.java | 13 ++ .../converter/RealtimeSegmentConverter.java| 3 + .../invertedindex/RealtimeLuceneTextIndex.java | 13 +- .../creator/impl/SegmentColumnarIndexCreator.java | 5 +- .../impl/SegmentIndexCreationDriverImpl.java | 45 +- .../creator/impl/text/LuceneTextIndexCreator.java | 153 - .../index/readers/text/LuceneTextIndexReader.java | 2 +- .../converter/RealtimeSegmentConverterTest.java| 153 - .../segment/store/FilePerIndexDirectoryTest.java | 11 +- .../store/SingleFileIndexDirectoryTest.java| 8 +- .../segment/spi/creator/IndexCreationContext.java | 48 ++- .../pinot/segment/spi/creator/SegmentCreator.java | 3 +- .../spi/creator/SegmentGeneratorConfig.java| 9 ++ .../pinot/segment/spi/index/TextIndexConfig.java | 2 +- .../segment/spi/index/mutable/MutableIndex.java| 8 ++ pom.xml| 2 +- 16 files changed, 443 insertions(+), 35 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump io.netty:netty-bom from 4.1.108.Final to 4.1.109.Final [pinot]
Jackie-Jiang merged PR #12929: URL: https://github.com/apache/pinot/pull/12929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/io.netty-netty-bom-4.1.109.Final deleted (was 598ec59b76)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch dependabot/maven/io.netty-netty-bom-4.1.109.Final in repository https://gitbox.apache.org/repos/asf/pinot.git was 598ec59b76 Bump io.netty:netty-bom from 4.1.108.Final to 4.1.109.Final The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated (2c6a84b8ec -> c823430f11)
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git from 2c6a84b8ec Improved segment build time for Lucene text index realtime to offline conversion (#12744) add c823430f11 Bump io.netty:netty-bom from 4.1.108.Final to 4.1.109.Final (#12929) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Refine PeerServerSegmentFinder [pinot]
Jackie-Jiang commented on code in PR #12933:
URL: https://github.com/apache/pinot/pull/12933#discussion_r1566428022
##
pinot-common/src/main/java/org/apache/pinot/core/util/PeerServerSegmentFinder.java:
##
@@ -47,93 +46,73 @@ public class PeerServerSegmentFinder {
private PeerServerSegmentFinder() {
}
- private static final Logger _logger =
LoggerFactory.getLogger(PeerServerSegmentFinder.class);
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PeerServerSegmentFinder.class);
private static final int MAX_NUM_ATTEMPTS = 5;
private static final int INITIAL_DELAY_MS = 500;
private static final double DELAY_SCALE_FACTOR = 2;
/**
- *
- * @param segmentName
- * @param downloadScheme Can be either http or https.
- * @param helixManager
- * @return a list of uri strings of the form
http(s)://hostname:port/segments/tablenameWithType/segmentName
- * for the servers hosting ONLINE segments; empty list if no such server
found.
+ * Returns a list of URIs of the form
'http(s)://hostname:port/segments/tableNameWithType/segmentName' for the servers
+ * hosting ONLINE segments; empty list if no such server found. The download
scheme can be either 'http' or 'https'.
*/
- public static List getPeerServerURIs(String segmentName, String
downloadScheme, HelixManager helixManager) {
-LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
-String tableNameWithType =
-
TableNameBuilder.forType(TableType.REALTIME).tableNameWithType(llcSegmentName.getTableName());
-return getPeerServerURIs(segmentName, downloadScheme, helixManager,
tableNameWithType);
- }
-
- public static List getPeerServerURIs(String segmentName, String
downloadScheme,
- HelixManager helixManager, String tableNameWithType) {
+ public static List getPeerServerURIs(HelixManager helixManager, String
tableNameWithType, String segmentName,
+ String downloadScheme) {
HelixAdmin helixAdmin = helixManager.getClusterManagmentTool();
String clusterName = helixManager.getClusterName();
-if (clusterName == null) {
- _logger.error("ClusterName not found");
- return ListUtils.EMPTY_LIST;
-}
-final List onlineServerURIs = new ArrayList<>();
+List onlineServerURIs = new ArrayList<>();
try {
RetryPolicies.exponentialBackoffRetryPolicy(MAX_NUM_ATTEMPTS,
INITIAL_DELAY_MS, DELAY_SCALE_FACTOR)
.attempt(() -> {
-getOnlineServersFromExternalView(segmentName, downloadScheme,
tableNameWithType, helixAdmin, clusterName,
+getOnlineServersFromExternalView(helixAdmin, clusterName,
tableNameWithType, segmentName, downloadScheme,
onlineServerURIs);
return !onlineServerURIs.isEmpty();
});
+} catch (AttemptsExceededException e) {
+ LOGGER.error("Failed to find ONLINE servers for segment: {}, table: {}",
segmentName, tableNameWithType);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Refine PeerServerSegmentFinder [pinot]
klsince commented on code in PR #12933:
URL: https://github.com/apache/pinot/pull/12933#discussion_r1566387588
##
pinot-common/src/main/java/org/apache/pinot/core/util/PeerServerSegmentFinder.java:
##
@@ -47,93 +46,73 @@ public class PeerServerSegmentFinder {
private PeerServerSegmentFinder() {
}
- private static final Logger _logger =
LoggerFactory.getLogger(PeerServerSegmentFinder.class);
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PeerServerSegmentFinder.class);
private static final int MAX_NUM_ATTEMPTS = 5;
private static final int INITIAL_DELAY_MS = 500;
private static final double DELAY_SCALE_FACTOR = 2;
/**
- *
- * @param segmentName
- * @param downloadScheme Can be either http or https.
- * @param helixManager
- * @return a list of uri strings of the form
http(s)://hostname:port/segments/tablenameWithType/segmentName
- * for the servers hosting ONLINE segments; empty list if no such server
found.
+ * Returns a list of URIs of the form
'http(s)://hostname:port/segments/tableNameWithType/segmentName' for the servers
+ * hosting ONLINE segments; empty list if no such server found. The download
scheme can be either 'http' or 'https'.
*/
- public static List getPeerServerURIs(String segmentName, String
downloadScheme, HelixManager helixManager) {
-LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
-String tableNameWithType =
-
TableNameBuilder.forType(TableType.REALTIME).tableNameWithType(llcSegmentName.getTableName());
-return getPeerServerURIs(segmentName, downloadScheme, helixManager,
tableNameWithType);
- }
-
- public static List getPeerServerURIs(String segmentName, String
downloadScheme,
- HelixManager helixManager, String tableNameWithType) {
+ public static List getPeerServerURIs(HelixManager helixManager, String
tableNameWithType, String segmentName,
+ String downloadScheme) {
HelixAdmin helixAdmin = helixManager.getClusterManagmentTool();
String clusterName = helixManager.getClusterName();
-if (clusterName == null) {
- _logger.error("ClusterName not found");
- return ListUtils.EMPTY_LIST;
-}
-final List onlineServerURIs = new ArrayList<>();
+List onlineServerURIs = new ArrayList<>();
try {
RetryPolicies.exponentialBackoffRetryPolicy(MAX_NUM_ATTEMPTS,
INITIAL_DELAY_MS, DELAY_SCALE_FACTOR)
.attempt(() -> {
-getOnlineServersFromExternalView(segmentName, downloadScheme,
tableNameWithType, helixAdmin, clusterName,
+getOnlineServersFromExternalView(helixAdmin, clusterName,
tableNameWithType, segmentName, downloadScheme,
onlineServerURIs);
return !onlineServerURIs.isEmpty();
});
+} catch (AttemptsExceededException e) {
+ LOGGER.error("Failed to find ONLINE servers for segment: {}, table: {}",
segmentName, tableNameWithType);
Review Comment:
nit: `"Failed to ... in {} attempts", ..., MAX_NUM_ATTEMPTS)` to make error
msg a bit more clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Refine PeerServerSegmentFinder [pinot]
Jackie-Jiang opened a new pull request, #12933: URL: https://github.com/apache/pinot/pull/12933 Separated from #12886 for easier review - Refine and simplify the logic of `PeerServerSegmentFinder` - Fix the bug of using netty port when admin port is not explicitly configured -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Followup fix and improvements on SchemaConformingTransformerV2 [pinot]
codecov-commenter commented on PR #12932: URL: https://github.com/apache/pinot/pull/12932#issuecomment-2057616056 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12932?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `0%` with `30 lines` in your changes are missing coverage. Please review. > Project coverage is 34.99%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`1e097f1`)](https://app.codecov.io/gh/apache/pinot/pull/12932?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 301 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/12932?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [...cordtransformer/SchemaConformingTransformerV2.java](https://app.codecov.io/gh/apache/pinot/pull/12932?src=pr=tree=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Frecordtransformer%2FSchemaConformingTransformerV2.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWNvcmR0cmFuc2Zvcm1lci9TY2hlbWFDb25mb3JtaW5nVHJhbnNmb3JtZXJWMi5qYXZh) | 0.00% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12932?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | | [...ingestion/SchemaConformingTransformerV2Config.java](https://app.codecov.io/gh/apache/pinot/pull/12932?src=pr=tree=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fconfig%2Ftable%2Fingestion%2FSchemaConformingTransformerV2Config.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL2luZ2VzdGlvbi9TY2hlbWFDb25mb3JtaW5nVHJhbnNmb3JtZXJWMkNvbmZpZy5qYXZh) | 0.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12932?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12932 +/- ## = - Coverage 61.75% 34.99% -26.76% = Files 2436 2428-8 Lines133233 132875 -358 Branches 2063620575 -61 = - Hits 8227446503-35771 - Misses4491182978+38067 + Partials 6048 3394 -2654 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12932/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12932/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12932/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12932/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12932/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12932/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-61.71%)` | :arrow_down: | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12932/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `34.99% <0.00%> (-26.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12932/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `34.99% <0.00%> (-26.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12932/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-27.73%)` | :arrow_down: | |
[PR] ReactPlayer, use Head from nextjs, move it above [pinot-site]
gio-startree opened a new pull request, #115: URL: https://github.com/apache/pinot-site/pull/115 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Try with ReactPlayer [pinot-site]
gio-startree closed pull request #114: Try with ReactPlayer URL: https://github.com/apache/pinot-site/pull/114 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Vector embeddings support in Pinot [pinot]
PeterCorless commented on issue #10919: URL: https://github.com/apache/pinot/issues/10919#issuecomment-2057576972 See [PR#11977](https://github.com/apache/pinot/pull/11977) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Followup fix and improvements on SchemaConformingTransformerV2 [pinot]
lnbest0707-uber opened a new pull request, #12932: URL: https://github.com/apache/pinot/pull/12932 tags: `feature` `bugfix` Followup from PR https://github.com/apache/pinot/pull/12788 with: - Getter/setter fix on SchemaConformingTransformerV2Config - Schema and config validation improvement - String type single value field ser/deser improvement - Column - JsonKeyPath mapping improvement - Unit test improvement with more comprehensive case coverage -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Isolate bad server configs during broker startup phase [pinot]
codecov-commenter commented on PR #12931: URL: https://github.com/apache/pinot/pull/12931#issuecomment-2057527084 ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/12931?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: Patch coverage is `0%` with `10 lines` in your changes are missing coverage. Please review. > Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`5a9a249`)](https://app.codecov.io/gh/apache/pinot/pull/12931?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 301 commits behind head on master. | [Files](https://app.codecov.io/gh/apache/pinot/pull/12931?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Patch % | Lines | |---|---|---| | [...che/pinot/broker/routing/BrokerRoutingManager.java](https://app.codecov.io/gh/apache/pinot/pull/12931?src=pr=tree=pinot-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fbroker%2Frouting%2FBrokerRoutingManager.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9Ccm9rZXJSb3V0aW5nTWFuYWdlci5qYXZh) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/12931?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #12931 +/- ## = - Coverage 61.75%0.00% -61.76% = Files 2436 2428-8 Lines133233 132856 -377 Branches 2063620567 -69 = - Hits 822740-82274 - Misses44911 132856+87945 + Partials 60480 -6048 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-61.63%)` | :arrow_down: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-61.75%)` | :arrow_down: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `0.00% <0.00%> (-61.76%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12931/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click
[PR] Isolate bad server configs during broker startup phase [pinot]
lnbest0707-uber opened a new pull request, #12931: URL: https://github.com/apache/pinot/pull/12931 Instructions: tags: `bugfix` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Try with ReactPlayer [pinot-site]
gio-startree opened a new pull request, #114: URL: https://github.com/apache/pinot-site/pull/114 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch master updated: Improved segment build time for Lucene text index realtime to offline conversion (#12744)
This is an automated email from the ASF dual-hosted git repository.
tingchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 2c6a84b8ec Improved segment build time for Lucene text index realtime
to offline conversion (#12744)
2c6a84b8ec is described below
commit 2c6a84b8ec840ed679c0a98230c518522b6591dd
Author: Christopher Peck <27231838+itschrisp...@users.noreply.github.com>
AuthorDate: Mon Apr 15 09:34:37 2024 -0700
Improved segment build time for Lucene text index realtime to offline
conversion (#12744)
* reuse mutable lucene index during segment conversion
* realtime segment conversion only
* add RealtimeSegmentConverter test for index reuse path
* clarify naming
* fix missed renaming
* address comments, close all resources
---
.../indexsegment/mutable/MutableSegmentImpl.java | 13 ++
.../converter/RealtimeSegmentConverter.java| 3 +
.../invertedindex/RealtimeLuceneTextIndex.java | 13 +-
.../creator/impl/SegmentColumnarIndexCreator.java | 5 +-
.../impl/SegmentIndexCreationDriverImpl.java | 45 +-
.../creator/impl/text/LuceneTextIndexCreator.java | 153 -
.../index/readers/text/LuceneTextIndexReader.java | 2 +-
.../converter/RealtimeSegmentConverterTest.java| 153 -
.../segment/store/FilePerIndexDirectoryTest.java | 11 +-
.../store/SingleFileIndexDirectoryTest.java| 8 +-
.../segment/spi/creator/IndexCreationContext.java | 48 ++-
.../pinot/segment/spi/creator/SegmentCreator.java | 3 +-
.../spi/creator/SegmentGeneratorConfig.java| 9 ++
.../pinot/segment/spi/index/TextIndexConfig.java | 2 +-
.../segment/spi/index/mutable/MutableIndex.java| 8 ++
15 files changed, 442 insertions(+), 34 deletions(-)
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java
index 55e0aec072..b336b30f20 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/mutable/MutableSegmentImpl.java
@@ -932,6 +932,19 @@ public class MutableSegmentImpl implements MutableSegment {
}
}
+ /**
+ * Calls commit() on all mutable indexes. This is used in preparation for
realtime segment conversion.
+ * .commit() can be implemented per index to perform any required actions
before using mutable segment
+ * artifacts to optimize immutable segment build.
+ */
+ public void commit() {
+for (IndexContainer indexContainer : _indexContainerMap.values()) {
+ for (MutableIndex mutableIndex :
indexContainer._mutableIndexes.values()) {
+mutableIndex.commit();
+ }
+}
+ }
+
@Override
public void destroy() {
_logger.info("Trying to close RealtimeSegmentImpl : {}", _segmentName);
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java
index ffb9bfc23f..0bf8fe571f 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java
@@ -118,6 +118,9 @@ public class RealtimeSegmentConverter {
genConfig.setNullHandlingEnabled(_nullHandlingEnabled);
genConfig.setSegmentZKPropsConfig(_segmentZKPropsConfig);
+// flush any artifacts to disk to improve mutable to immutable segment
conversion
+_realtimeSegmentImpl.commit();
+
SegmentIndexCreationDriverImpl driver = new
SegmentIndexCreationDriverImpl();
try (PinotSegmentRecordReader recordReader = new
PinotSegmentRecordReader()) {
int[] sortedDocIds = _columnIndicesForRealtimeTable.getSortedColumn() !=
null
diff --git
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java
index a71d2663ed..8d2e43c8a5 100644
---
a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java
+++
b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java
@@ -78,7 +78,7 @@ public class RealtimeLuceneTextIndex implements
MutableTextIndex {
// for realtime
_indexCreator =
new
Re: [PR] Improved segment build time for Lucene text index realtime to offline conversion [pinot]
chenboat merged PR #12744: URL: https://github.com/apache/pinot/pull/12744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Skip node level throttle during server bootstrap - Enhancement [pinot]
deepthi912 opened a new pull request, #12930: URL: https://github.com/apache/pinot/pull/12930 Fix for the following issue: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump slf4j.version from 2.0.12 to 2.0.13 [pinot]
sullis commented on PR #12928: URL: https://github.com/apache/pinot/pull/12928#issuecomment-2056796309 LGTM https://www.slf4j.org/news.html https://github.com/apache/pinot/assets/30938/9518dd06-c722-475a-bd32-6dbfa823fef8;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Bump io.netty:netty-bom from 4.1.108.Final to 4.1.109.Final [pinot]
sullis commented on PR #12929: URL: https://github.com/apache/pinot/pull/12929#issuecomment-2056794602 LGTM https://netty.io/news/2024/04/15/4-1-109-Final.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/io.netty-netty-bom-4.1.109.Final created (now 598ec59b76)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/io.netty-netty-bom-4.1.109.Final in repository https://gitbox.apache.org/repos/asf/pinot.git at 598ec59b76 Bump io.netty:netty-bom from 4.1.108.Final to 4.1.109.Final No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump io.netty:netty-bom from 4.1.108.Final to 4.1.109.Final [pinot]
dependabot[bot] opened a new pull request, #12929: URL: https://github.com/apache/pinot/pull/12929 Bumps [io.netty:netty-bom](https://github.com/netty/netty) from 4.1.108.Final to 4.1.109.Final. Commits https://github.com/netty/netty/commit/43455dfe98202b8dc63d89ac3f04027c3e8f9f54;>43455df [maven-release-plugin] prepare release netty-4.1.109.Final https://github.com/netty/netty/commit/b9af2e686a5e9e2549a85bcfb9d9fe4e979475f7;>b9af2e6 Utilize ByteBuf#indexOf (https://redirect.github.com/netty/netty/issues/13974;>#13974) https://github.com/netty/netty/commit/9f44b970708045aa19970cb417ec7127c2db2c17;>9f44b97 Fix DefaultChannelId#asLongText NPE (https://redirect.github.com/netty/netty/issues/13971;>#13971) https://github.com/netty/netty/commit/eb48822ed035653eb4fdc166b15734d7b8cb633e;>eb48822 chore: remove repetitive words (https://redirect.github.com/netty/netty/issues/13972;>#13972) https://github.com/netty/netty/commit/0e436dd4504918a6d19a0973fc21bfcb02194418;>0e436dd Use PlatformDependent.threadLocalRandom() to reduce memory footprint (https://redirect.github.com/netty/netty/issues/13977;>#13977) https://github.com/netty/netty/commit/4d961d09145f715e0f3d34f0ecddc973d351afcd;>4d961d0 Don't send a RST frame when closing the stream in a write future whil… (https://redirect.github.com/netty/netty/issues/13973;>#13973) https://github.com/netty/netty/commit/e19c91f73aa9d9ae2e67c0b01847f72a882196d1;>e19c91f Improve new DefaultChannelId (https://redirect.github.com/netty/netty/issues/13960;>#13960) https://github.com/netty/netty/commit/02b54ca54b63170c20ddbdae3c04621518a8e8f3;>02b54ca Refactor file descriptor handling in openFileDescriptors() (https://redirect.github.com/netty/netty/issues/13955;>#13955) https://github.com/netty/netty/commit/33ed825d5bc1608cc9b6284a7238619e53c09e02;>33ed825 Fix voidPromise in Http2FrameCodec.writeHeadersFrame (https://redirect.github.com/netty/netty/issues/13958;>#13958) https://github.com/netty/netty/commit/a38a85c908f41f03276e795851ee931bcb16dc49;>a38a85c Improve ByteBufUtil#lastIndexOf (https://redirect.github.com/netty/netty/issues/13942;>#13942) Additional commits viewable in https://github.com/netty/netty/compare/netty-4.1.108.Final...netty-4.1.109.Final;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump slf4j.version from 2.0.12 to 2.0.13 [pinot]
dependabot[bot] opened a new pull request, #12928: URL: https://github.com/apache/pinot/pull/12928 Bumps `slf4j.version` from 2.0.12 to 2.0.13. Updates `org.slf4j:jcl-over-slf4j` from 2.0.12 to 2.0.13 Updates `org.slf4j:slf4j-api` from 2.0.12 to 2.0.13 Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.apache.maven.scm-maven-scm-provider-gitexe-2.1.0 created (now 068abcfc4d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.maven.scm-maven-scm-provider-gitexe-2.1.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at 068abcfc4d Bump org.apache.maven.scm:maven-scm-provider-gitexe from 2.0.1 to 2.1.0 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/org.apache.maven.plugins-maven-jar-plugin-3.4.0 created (now 3130039947)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.maven.plugins-maven-jar-plugin-3.4.0 in repository https://gitbox.apache.org/repos/asf/pinot.git at 3130039947 Bump org.apache.maven.plugins:maven-jar-plugin from 3.3.0 to 3.4.0 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/aws.sdk.version-2.25.31 created (now 5863c220e8)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/aws.sdk.version-2.25.31 in repository https://gitbox.apache.org/repos/asf/pinot.git at 5863c220e8 Bump aws.sdk.version from 2.25.30 to 2.25.31 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(pinot) branch dependabot/maven/slf4j.version-2.0.13 created (now 6c539cc661)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/slf4j.version-2.0.13 in repository https://gitbox.apache.org/repos/asf/pinot.git at 6c539cc661 Bump slf4j.version from 2.0.12 to 2.0.13 No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump org.apache.maven.plugins:maven-jar-plugin from 3.3.0 to 3.4.0 [pinot]
dependabot[bot] opened a new pull request, #12927: URL: https://github.com/apache/pinot/pull/12927 Bumps [org.apache.maven.plugins:maven-jar-plugin](https://github.com/apache/maven-jar-plugin) from 3.3.0 to 3.4.0. Release notes Sourced from https://github.com/apache/maven-jar-plugin/releases;>org.apache.maven.plugins:maven-jar-plugin's releases. 3.4.0 New features and improvements https://issues.apache.org/jira/browse/MJAR-296;>[MJAR-296] - Allow including files excluded by default. (https://redirect.github.com/apache/maven-jar-plugin/pull/67;>#67) https://github.com/redzi;>@redzi https://issues.apache.org/jira/browse/MJAR-302;>[MJAR-302] - Require Maven 3.6.3 (https://redirect.github.com/apache/maven-jar-plugin/pull/77;>#77) https://github.com/slawekjaranowski;>@slawekjaranowski https://issues.apache.org/jira/browse/MJAR-292;>[MJAR-292] - Detect MRJAR and add Multi-Release manifest entry (https://redirect.github.com/apache/maven-jar-plugin/pull/57;>#57) https://github.com/jorsol;>@jorsol Bug Fixes https://issues.apache.org/jira/browse/MJAR-62;>[MJAR-62] - Set Build-Jdk according to used toolchain (https://redirect.github.com/apache/maven-jar-plugin/pull/73;>#73) https://github.com/slawekjaranowski;>@slawekjaranowski Dependency updates https://issues.apache.org/jira/browse/MJAR-306;>[MJAR-306] - Use properties for plugins versions in LifecycleMapping (https://redirect.github.com/apache/maven-jar-plugin/pull/82;>#82) https://github.com/slawekjaranowski;>@slawekjaranowski Bump org.junit:junit-bom from 5.10.1 to 5.10.2 (https://redirect.github.com/apache/maven-jar-plugin/pull/74;>#74) https://github.com/dependabot;>@dependabot https://issues.apache.org/jira/browse/MJAR-298;>[MJAR-298] - Update Maven-Archiver to 3.6.2 (https://redirect.github.com/apache/maven-jar-plugin/pull/78;>#78) https://github.com/slawekjaranowski;>@slawekjaranowski Bump apache/maven-gh-actions-shared from 2 to 4 (https://redirect.github.com/apache/maven-jar-plugin/pull/72;>#72) https://github.com/dependabot;>@dependabot Bump commons-io:commons-io from 2.14.0 to 2.15.1 (https://redirect.github.com/apache/maven-jar-plugin/pull/68;>#68) https://github.com/dependabot;>@dependabot https://issues.apache.org/jira/browse/MJAR-297;>[MJAR-297] - Update Parent to 40 (https://redirect.github.com/apache/maven-jar-plugin/pull/64;>#64) https://github.com/jorsol;>@jorsol https://issues.apache.org/jira/browse/MJAR-298;>[MJAR-298] - Update Maven-Archiver to 3.6.1 (https://redirect.github.com/apache/maven-jar-plugin/pull/65;>#65) https://github.com/jorsol;>@jorsol https://issues.apache.org/jira/browse/MJAR-293;>[MJAR-293] - Update Parent to 39 (https://redirect.github.com/apache/maven-jar-plugin/pull/59;>#59) https://github.com/jorsol;>@jorsol Maintenance https://issues.apache.org/jira/browse/MJAR-304;>[MJAR-304] - Refresh download page (https://redirect.github.com/apache/maven-jar-plugin/pull/80;>#80) https://github.com/slawekjaranowski;>@slawekjaranowski https://issues.apache.org/jira/browse/MJAR-303;>[MJAR-303] - Cleanup declared dependencies (https://redirect.github.com/apache/maven-jar-plugin/pull/79;>#79) https://github.com/slawekjaranowski;>@slawekjaranowski Remove dependency on plexus (https://redirect.github.com/apache/maven-jar-plugin/pull/63;>#63) https://github.com/elharo;>@elharo Commits https://github.com/apache/maven-jar-plugin/commit/992f44a2ef710326a6b98e18d4e40f724bea5f90;>992f44a [maven-release-plugin] prepare release maven-jar-plugin-3.4.0 https://github.com/apache/maven-jar-plugin/commit/5e31b9933a68a44b180a499636f7f43c8521648c;>5e31b99 [MJAR-296] Allow including files excluded by default. (https://redirect.github.com/apache/maven-jar-plugin/issues/67;>#67) https://github.com/apache/maven-jar-plugin/commit/ddfb635b55ec1ad23f83c4884a743b4a6c80a273;>ddfb635 [MJAR-306] Use properties for plugins versions in LifecycleMapping https://github.com/apache/maven-jar-plugin/commit/aeffa392dd83040cde908c633b0ce2406850193b;>aeffa39 [MJAR-304] Refresh download page https://github.com/apache/maven-jar-plugin/commit/ee85d599e14cbc69b71c65ade07b57f616217b57;>ee85d59 [MJAR-303] Cleanup declared dependencies https://github.com/apache/maven-jar-plugin/commit/845c12071f2ec733c985c3d0eab3e48b1c15a486;>845c120 Bump org.junit:junit-bom from 5.10.1 to 5.10.2 https://github.com/apache/maven-jar-plugin/commit/8dd0d3f176fb28aa779cd2ddec9c342e834f38c6;>8dd0d3f [MJAR-298] Update Maven-Archiver to 3.6.2 https://github.com/apache/maven-jar-plugin/commit/1b958d1804fa894eb4838c13b0532742751e5836;>1b958d1 [MJAR-302] Require Maven 3.6.3 https://github.com/apache/maven-jar-plugin/commit/fa4330fa687a98b1fa43c9bc8a574408b2ffe40b;>fa4330f [MJAR-62] Set Build-Jdk according to used toolchain
[PR] Bump aws.sdk.version from 2.25.30 to 2.25.31 [pinot]
dependabot[bot] opened a new pull request, #12926: URL: https://github.com/apache/pinot/pull/12926 Bumps `aws.sdk.version` from 2.25.30 to 2.25.31. Updates `software.amazon.awssdk:bom` from 2.25.30 to 2.25.31 Updates `software.amazon.awssdk:apache-client` from 2.25.30 to 2.25.31 Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[PR] Bump org.apache.maven.scm:maven-scm-provider-gitexe from 2.0.1 to 2.1.0 [pinot]
dependabot[bot] opened a new pull request, #12925: URL: https://github.com/apache/pinot/pull/12925 Bumps org.apache.maven.scm:maven-scm-provider-gitexe from 2.0.1 to 2.1.0. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Compatibility regression testing against Pinot release 1.1.0 [pinot]
abhioncbr commented on issue #12887: URL: https://github.com/apache/pinot/issues/12887#issuecomment-2056492252 Changes merged. closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Compatibility regression testing against Pinot release 1.1.0 [pinot]
abhioncbr closed issue #12887: Compatibility regression testing against Pinot release 1.1.0 URL: https://github.com/apache/pinot/issues/12887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565545708
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/MultiStageQueryStats.java:
##
@@ -0,0 +1,668 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.runtime.plan;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.google.common.base.Preconditions;
+import java.io.DataInput;
+import java.io.DataInputStream;
+import java.io.DataOutput;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.UncheckedIOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+import org.apache.avro.util.ByteBufferInputStream;
+import org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.query.runtime.operator.BaseMailboxReceiveOperator;
+import
org.apache.pinot.query.runtime.operator.LeafStageTransferableBlockOperator;
+import org.apache.pinot.query.runtime.operator.LiteralValueOperator;
+import org.apache.pinot.query.runtime.operator.MailboxSendOperator;
+import org.apache.pinot.query.runtime.operator.MultiStageOperator;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The stats of a given query.
+ *
+ * For the same query, multiple instances of this class may exist. Each of
them will have a partial view of the stats.
+ * Specifically, while the query is being executed, each operator will return
its own partial view of the stats when
+ * EOS block is sent.
+ *
+ * Simple operations with a single upstream, like filters or transforms, would
just add their own information to the
+ * stats. More complex operations, like joins or receiving mailboxes, will
merge the stats from all their upstreams and
+ * add their own stats.
+ *
+ * The complete stats for the query are obtained in the execution root
(usually the broker) by merging the partial
+ * views.
+ *
+ * In order to reduce allocation, this class is mutable. Some operators may
create their own stats, but most of them
+ * will receive a stats object from the upstream operator and modify it by
adding their own stats and sometimes merging
+ * them with other upstream stats.
+ */
+public class MultiStageQueryStats {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(MultiStageQueryStats.class);
+ private final int _currentStageId;
+ private final StageStats.Open _currentStats;
+ /**
+ * Known stats for stages whose id is higher than the current one.
+ *
+ * A stage may not know all the stats whose id is higher than the current
one, so this list may contain null values.
+ * It may also grow in size when different merge methods are called.
+ *
+ * For example the stats of the left hand side of a join may know stats of
stages 3 and 4 and the right side may know
+ * stats of stages 5. When merging the stats of the join, the stats of
stages 5 will be added to this list.
+ *
+ * @see #mergeUpstream(List)
+ * @see #mergeUpstream(MultiStageQueryStats)
+ * @see #mergeInOrder(MultiStageQueryStats, MultiStageOperator.Type, StatMap)
+ */
+ private final ArrayList _closedStats;
+ private static final MultiStageOperator.Type[] ALL_TYPES =
MultiStageOperator.Type.values();
+
+ private MultiStageQueryStats(int stageId) {
+_currentStageId = stageId;
+_currentStats = new StageStats.Open();
+_closedStats = new ArrayList<>();
+ }
+
+ private static MultiStageQueryStats create(int stageId,
MultiStageOperator.Type type, @Nullable StatMap opStats) {
+MultiStageQueryStats multiStageQueryStats = new
MultiStageQueryStats(stageId);
+
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565537934
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/utils/BlockingMultiStreamConsumer.java:
##
@@ -240,6 +253,15 @@ protected boolean isEos(TransferableBlock element) {
return element.isSuccessfulEndOfStreamBlock();
}
+@Override
+protected void onConsumerFinish(TransferableBlock element) {
+ if (element.getQueryStats() != null) {
+_stats.mergeUpstream(element.getQueryStats());
+ } else {
+_stats.mergeUpstream(element.getSerializedStatsByStage());
Review Comment:
By doing this we don't need to deserialize the stats when they are sent
using memory mailboxes. But right now we are not prioritizing sending the stats
there. That is something we can improve in future PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565537934
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/utils/BlockingMultiStreamConsumer.java:
##
@@ -240,6 +253,15 @@ protected boolean isEos(TransferableBlock element) {
return element.isSuccessfulEndOfStreamBlock();
}
+@Override
+protected void onConsumerFinish(TransferableBlock element) {
+ if (element.getQueryStats() != null) {
+_stats.mergeUpstream(element.getQueryStats());
+ } else {
+_stats.mergeUpstream(element.getSerializedStatsByStage());
Review Comment:
By doing this we don't need to deserialize the stats when they are sent
using memory mailboxes. But right now we are not prioritizing sending the stats
there. That is something we can improve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565423170
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultiStageOperator.java:
##
@@ -19,107 +19,295 @@
package org.apache.pinot.query.runtime.operator;
import com.google.common.base.Joiner;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Stopwatch;
+import java.io.DataInput;
+import java.io.IOException;
import java.util.List;
+import java.util.concurrent.TimeUnit;
+import org.apache.pinot.common.datatable.DataTable;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.common.response.broker.BrokerResponseNativeV2;
import org.apache.pinot.core.common.Operator;
import org.apache.pinot.query.runtime.blocks.TransferableBlock;
import org.apache.pinot.query.runtime.blocks.TransferableBlockUtils;
+import org.apache.pinot.query.runtime.plan.MultiStageQueryStats;
import org.apache.pinot.query.runtime.plan.OpChainExecutionContext;
+import org.apache.pinot.query.runtime.plan.pipeline.PipelineBreakerOperator;
import org.apache.pinot.spi.exception.EarlyTerminationException;
import org.apache.pinot.spi.trace.InvocationScope;
import org.apache.pinot.spi.trace.Tracing;
import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-public abstract class MultiStageOperator implements
Operator, AutoCloseable {
- private static final Logger LOGGER =
LoggerFactory.getLogger(MultiStageOperator.class);
+public abstract class MultiStageOperator & StatMap.Key>
+implements Operator, AutoCloseable {
protected final OpChainExecutionContext _context;
protected final String _operatorId;
- protected final OpChainStats _opChainStats;
+ protected final StatMap _statMap;
protected boolean _isEarlyTerminated;
- public MultiStageOperator(OpChainExecutionContext context) {
+ public MultiStageOperator(OpChainExecutionContext context, Class
keyStatClass) {
_context = context;
_operatorId = Joiner.on("_").join(getClass().getSimpleName(),
_context.getStageId(), _context.getServer());
-_opChainStats = _context.getStats();
_isEarlyTerminated = false;
+_statMap = new StatMap<>(keyStatClass);
}
+ protected abstract Logger logger();
+
+ public abstract Type getOperatorType();
+
+ public abstract K getExecutionTimeKey();
+
+ public abstract K getEmittedRowsKey();
+
@Override
public TransferableBlock nextBlock() {
if (Tracing.ThreadAccountantOps.isInterrupted()) {
throw new EarlyTerminationException("Interrupted while processing next
block");
}
+if (logger().isDebugEnabled()) {
+ logger().debug("Operator {}: Reading next block", _operatorId);
+}
try (InvocationScope ignored =
Tracing.getTracer().createScope(getClass())) {
TransferableBlock nextBlock;
if (shouldCollectStats()) {
-OperatorStats operatorStats = _opChainStats.getOperatorStats(_context,
_operatorId);
-operatorStats.startTimer();
+Stopwatch executeStopwatch = Stopwatch.createStarted();
Review Comment:
We discussed on whether we want to always collect time stats or not. Right
now this PR is not modifying that, but it should be as easy as to always return
true in `shouldCollectStats` or just remove this if
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565400481
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/AggregateOperator.java:
##
@@ -132,29 +156,31 @@ public String toExplainString() {
@Override
protected TransferableBlock getNextBlock() {
if (_hasConstructedAggregateBlock) {
- return TransferableBlockUtils.getEndOfStreamTransferableBlock();
+ assert _eosBlock != null;
Review Comment:
Instead of an empty EOS block now we send a block with all the stats. This
is a pattern that is repeated in all other operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565397287
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/MailboxContentObserver.java:
##
@@ -88,6 +71,8 @@ public void onNext(MailboxContent mailboxContent) {
LOGGER.warn("Mailbox: {} already cancelled from upstream",
mailboxId);
cancelStream();
break;
+case FIRST_ERROR:
+ return;
Review Comment:
I added this enum to keep the same behavior in this class, which just
aborted the execution when exceptions are found in a message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565395943
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/channel/MailboxContentObserver.java:
##
@@ -62,22 +58,9 @@ public void onNext(MailboxContent mailboxContent) {
_mailbox = _mailboxService.getReceivingMailbox(mailboxId);
}
try {
- TransferableBlock block;
- DataBlock dataBlock =
DataBlockUtils.getDataBlock(mailboxContent.getPayload().asReadOnlyByteBuffer());
- if (dataBlock instanceof MetadataBlock) {
-Map exceptions = dataBlock.getExceptions();
-if (exceptions.isEmpty()) {
- block =
TransferableBlockUtils.getEndOfStreamTransferableBlock(((MetadataBlock)
dataBlock).getStats());
-} else {
-
_mailbox.setErrorBlock(TransferableBlockUtils.getErrorTransferableBlock(exceptions));
- return;
-}
- } else {
-block = new TransferableBlock(dataBlock);
- }
-
Review Comment:
This code has been moved to `ReceivingMailbox.offerRaw` in order to be able
to capture the stats there. Also, I think the responsibility of doing this
should be in ReceivingMailbox and not in the pipeline itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1565391241 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/LeafStageTransferableBlockOperator.java: ## @@ -99,10 +105,32 @@ public LeafStageTransferableBlockOperator(OpChainExecutionContext context, List< Integer maxStreamingPendingBlocks = QueryOptionsUtils.getMaxStreamingPendingBlocks(context.getOpChainMetadata()); _blockingQueue = new ArrayBlockingQueue<>(maxStreamingPendingBlocks != null ? maxStreamingPendingBlocks : QueryOptionValue.DEFAULT_MAX_STREAMING_PENDING_BLOCKS); +String tableName = context.getLeafStageContext().getStagePlan().getStageMetadata().getTableName(); +_statMap.merge(StatKey.TABLE, tableName); Review Comment: This is one of the places where we are modifying the map from different threads. Here the constructor is called by a different thread than the one executing the code. In fact I found the concurrency issue in leaf stages stats. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565387972
##
pinot-query-runtime/src/main/java/org/apache/pinot/query/mailbox/GrpcSendingMailbox.java:
##
@@ -64,25 +69,31 @@ public void send(TransferableBlock block)
return;
}
if (LOGGER.isDebugEnabled()) {
- LOGGER.debug("==[GRPC SEND]== sending data to: " + _id);
+ LOGGER.debug("==[GRPC SEND]== sending message " + block + " to: " + _id);
}
if (_contentObserver == null) {
_contentObserver = getContentObserver();
}
_contentObserver.onNext(toMailboxContent(block));
+if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug("==[GRPC SEND]== message " + block + " sent to: " + _id);
+}
Review Comment:
I've found this logs very useful to debug a concurrency issue in this PR.
All these logs are either not in the hot path or warded with `isDebugEnabled`
in order to do not allocate if disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565385608
##
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotJoinToDynamicBroadcastRule.java:
##
@@ -164,11 +165,15 @@ public void onMatch(RelOptRuleCall call) {
//2. when hash key are the same but hash functions are different, it
can be done via normal hash shuffle.
boolean isColocatedJoin =
PinotHintStrategyTable.isHintOptionTrue(join.getHints(),
PinotHintOptions.JOIN_HINT_OPTIONS,
PinotHintOptions.JoinHintOptions.IS_COLOCATED_BY_JOIN_KEYS);
-PinotLogicalExchange dynamicBroadcastExchange = isColocatedJoin
-? PinotLogicalExchange.create(right.getInput(),
RelDistributions.hash(join.analyzeCondition().rightKeys),
-PinotRelExchangeType.PIPELINE_BREAKER)
-: PinotLogicalExchange.create(right.getInput(),
RelDistributions.BROADCAST_DISTRIBUTED,
-PinotRelExchangeType.PIPELINE_BREAKER);
+PinotLogicalExchange dynamicBroadcastExchange;
+RelNode rightInput = right.getInput();
+if (isColocatedJoin) {
+ RelDistribution dist =
RelDistributions.hash(join.analyzeCondition().rightKeys);
+ dynamicBroadcastExchange = PinotLogicalExchange.create(rightInput, dist,
PinotRelExchangeType.PIPELINE_BREAKER);
+} else {
+ RelDistribution dist = RelDistributions.BROADCAST_DISTRIBUTED;
+ dynamicBroadcastExchange = PinotLogicalExchange.create(rightInput, dist,
PinotRelExchangeType.PIPELINE_BREAKER);
+}
Review Comment:
This is the same code as before, but easier to read and to debug. The former
is subjective, so I'm open to rollback this change if required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565384127
##
pinot-core/src/main/java/org/apache/pinot/core/query/reduce/ExecutionStatsAggregator.java:
##
@@ -364,25 +357,6 @@ public void setStats(@Nullable String rawTableName,
BrokerResponseNative brokerR
}
}
- public void setStageLevelStats(@Nullable String rawTableName,
BrokerResponseStats brokerResponseStats,
- @Nullable BrokerMetrics brokerMetrics) {
-if (_enableTrace) {
- setStats(rawTableName, brokerResponseStats, brokerMetrics);
- brokerResponseStats.setOperatorStats(_operatorStats);
-}
-
-brokerResponseStats.setNumBlocks(_numBlocks);
-brokerResponseStats.setNumRows(_numRows);
-brokerResponseStats.setMaxRowsInJoinReached(_maxRowsInJoinReached);
-brokerResponseStats.setNumGroupsLimitReached(_numGroupsLimitReached);
-brokerResponseStats.setStageExecutionTimeMs(_stageExecutionTimeMs);
-brokerResponseStats.setStageExecutionUnit(_stageExecutionUnit);
-brokerResponseStats.setTableNames(new ArrayList<>(_tableNames));
-if (_stageExecStartTimeMs >= 0 && _stageExecEndTimeMs >= 0) {
- brokerResponseStats.setStageExecWallTimeMs(_stageExecEndTimeMs -
_stageExecStartTimeMs);
-}
- }
-
Review Comment:
This method is not used anymore. Instead we merge stats in
`MultiStageOperator.Type.mergeInto`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565379309
##
pinot-common/src/main/java/org/apache/pinot/common/response/broker/BrokerResponseNativeV2.java:
##
@@ -61,36 +67,31 @@ public BrokerResponseNativeV2(List
exceptions) {
super(exceptions);
}
- /** Generate EXPLAIN PLAN output when queries are evaluated by Broker
without going to the Server. */
- private static BrokerResponseNativeV2 getBrokerResponseExplainPlanOutput() {
-BrokerResponseNativeV2 brokerResponse = BrokerResponseNativeV2.empty();
-List rows = new ArrayList<>();
-rows.add(new Object[]{"BROKER_EVALUATE", 0, -1});
-brokerResponse.setResultTable(new
ResultTable(DataSchema.EXPLAIN_RESULT_SCHEMA, rows));
-return brokerResponse;
Review Comment:
This method was not used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565358229
##
pinot-common/src/main/java/org/apache/pinot/common/datablock/DataBlockUtils.java:
##
@@ -62,13 +67,13 @@ public static MetadataBlock getEndOfStreamDataBlock() {
return new MetadataBlock(MetadataBlock.MetadataBlockType.EOS);
}
- public static MetadataBlock getEndOfStreamDataBlock(Map
stats) {
-return new MetadataBlock(MetadataBlock.MetadataBlockType.EOS, stats);
+ public static int readVersionType(ByteBuffer byteBuffer) {
+return byteBuffer.getInt();
Review Comment:
This method is here to be called from a test, but can be useful in other
cases in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565356393
##
pinot-common/src/main/java/org/apache/pinot/common/datablock/BaseDataBlock.java:
##
@@ -572,14 +585,9 @@ private Map
deserializeExceptions(ByteBuffer buffer)
@Override
public String toString() {
if (_dataSchema == null) {
- return _metadata.toString();
+ return "{}";
Review Comment:
We don't have the ability to decode what is in metadata now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1565315967 ## pinot-broker/src/test/java/org/apache/pinot/broker/querylog/QueryLoggerTest.java: ## @@ -259,11 +259,9 @@ private QueryLogger.QueryLogParams generateParams(boolean isGroupLimitHit, int n response.setExceptions( IntStream.range(0, numExceptions) .mapToObj(i -> new ProcessingException()).collect(Collectors.toList())); -response.setOfflineTotalCpuTimeNs(14); Review Comment: BTW, this is why in lines 110 an 111 I had to modify the result. Before we were returning the synthetic values 14 and 18 and now we return 48 and 60 which is the sum of 15, 16 and 17 and 19, 20 and 21 respectively. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1565313954 ## pinot-broker/src/test/java/org/apache/pinot/broker/querylog/QueryLoggerTest.java: ## @@ -259,11 +259,9 @@ private QueryLogger.QueryLogParams generateParams(boolean isGroupLimitHit, int n response.setExceptions( IntStream.range(0, numExceptions) .mapToObj(i -> new ProcessingException()).collect(Collectors.toList())); -response.setOfflineTotalCpuTimeNs(14); Review Comment: Yes, I've removed these `setOflineTotalCpuTimeNs` and `setRealtimeTotalCpuTimeNs` from `BrokerResponseNative`. In production code they are always set as `threadCpuTimeNs + systemActivitiesCpuTimeNs + responseSerializationCpuTimeNs`, so it looks error prone to have a setter (like the one we used here). Instead we never set the value of the total and always derive it from its parts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704: URL: https://github.com/apache/pinot/pull/12704#discussion_r1565308960 ## pinot-common/src/main/java/org/apache/pinot/common/datablock/DataBlock.java: ## @@ -80,8 +80,6 @@ byte[] toBytes() @Nullable RoaringBitmap getNullRowIds(int colId); - DataBlock toMetadataOnlyDataTable(); - DataBlock toDataOnlyDataTable(); Review Comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565307236
##
pinot-common/src/main/java/org/apache/pinot/common/datablock/MetadataBlock.java:
##
@@ -36,108 +39,129 @@
*/
public class MetadataBlock extends BaseDataBlock {
- private static final ObjectMapper JSON = new ObjectMapper();
-
+ private static final Logger LOGGER =
LoggerFactory.getLogger(MetadataBlock.class);
@VisibleForTesting
- static final int VERSION = 1;
+ static final int VERSION = 2;
Review Comment:
I think it is cleaner to read a different version and explicitly write in
the code what we do when a block of an older version is received. Instead of
crossing our fingers knowing that v1 uses varsize while v2 uses metadata, now
in `MetadataBlock.deserialize` we an explicit code saying what we are doing.
I'm open to increase the version of the other blocks as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [PR] Multi stage stats [pinot]
gortiz commented on code in PR #12704:
URL: https://github.com/apache/pinot/pull/12704#discussion_r1565303435
##
pinot-common/src/main/java/org/apache/pinot/common/datatable/DataTable.java:
##
@@ -100,7 +110,7 @@ enum MetadataValueType {
* - NEVER decrease MAX_ID
* Otherwise, backward compatibility will be broken.
*/
- enum MetadataKey {
+ enum MetadataKey implements StatMap.Key {
Review Comment:
These are not he stats reported by Leaf operator. This operator uses its own
`LeafStageTransferableBlockOperator.StatKey` class. But it is useful to
implement `StatMap.Key` in `MetadataKey` in order to be able to store them in a
new `StatMap`. Do we use that in V1? No, but in V2 we use it in order to:
* In the leaf operator, but only as an input. Leaf operator has its own stat
class, but it receives MetadataKey stats fromt he V1 engine in a `Map` and has to transform them into its own stats. See
`LeafStageTransferableBlockOperator.mergeExecutionStats`.
* In brokers, when the response is being reduced. I've changed that code to
reduce it using StatMap instead of a `HashMap`. Specifically,
I've removed a lot of attributes from `BrokerResponseNative` and transform them
into a `StatMap _serverStats`. This makes it easier to
merge V2 stats into this object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Use Foreign Memory API, introduced in Java 22 [pinot]
gortiz commented on issue #12809: URL: https://github.com/apache/pinot/issues/12809#issuecomment-2055654108 I think there is a third part which may be more complex: How to compile a maven subproject that requires Java 22 while not affecting the rest of the project (which requires Java 11 or 8). About point A, I already created one implementation at the same time I've created the unsafe buffers, but ended up removing it to do not have to deal with the third point I mentioned. The code is already in git (see for example [this commit](https://github.com/gortiz/pinot/tree/dc227182efaebff29e12320f5ba833ad269f6a54/pinot-plugins/pinot-foreign-memory/src/main/java/org/apache/pinot/foreign/memory)). Feel free to get inspiration from it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
