[GitHub] [incubator-pinot] codecov-io commented on issue #5148: Fix a formatting bug in AggregationFunctionUtils
codecov-io commented on issue #5148: Fix a formatting bug in AggregationFunctionUtils URL: https://github.com/apache/incubator-pinot/pull/5148#issuecomment-598524343 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5148?src=pr=h1) Report > Merging [#5148](https://codecov.io/gh/apache/incubator-pinot/pull/5148?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/6b6fa56ab75e5f59f58249f7a733a375421b94b6?src=pr=desc) will **increase** coverage by `9.1%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-pinot/pull/5148?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#5148 +/- ## === + Coverage 56.5% 65.61% +9.1% Complexity 12 12 === Files 1052 1052 Lines 5413654137 +1 Branches 8080 8081 +1 === + Hits 3059035522 +4932 + Misses2110315985 -5118 - Partials 2443 2630+187 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5148?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...aggregation/function/AggregationFunctionUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9BZ2dyZWdhdGlvbkZ1bmN0aW9uVXRpbHMuamF2YQ==) | `95.23% <100%> (+2.55%)` | `0 <0> (ø)` | :arrow_down: | | [...time/impl/dictionary/OffHeapMutableBytesStore.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvT2ZmSGVhcE11dGFibGVCeXRlc1N0b3JlLmphdmE=) | `91.11% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...pql/parsers/PinotQuery2BrokerRequestConverter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9wcWwvcGFyc2Vycy9QaW5vdFF1ZXJ5MkJyb2tlclJlcXVlc3RDb252ZXJ0ZXIuamF2YQ==) | `88.07% <0%> (+0.45%)` | `0% <0%> (ø)` | :arrow_down: | | [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `88.25% <0%> (+0.75%)` | `0% <0%> (ø)` | :arrow_down: | | [.../helix/core/realtime/SegmentCompletionManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1NlZ21lbnRDb21wbGV0aW9uTWFuYWdlci5qYXZh) | `70.63% <0%> (+0.85%)` | `0% <0%> (ø)` | :arrow_down: | | [...java/org/apache/pinot/common/utils/DataSchema.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVNjaGVtYS5qYXZh) | `85.26% <0%> (+1.05%)` | `0% <0%> (ø)` | :arrow_down: | | [.../FixedByteSingleColumnSingleValueReaderWriter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pby9yZWFkZXJ3cml0ZXIvaW1wbC9GaXhlZEJ5dGVTaW5nbGVDb2x1bW5TaW5nbGVWYWx1ZVJlYWRlcldyaXRlci5qYXZh) | `98.94% <0%> (+1.05%)` | `0% <0%> (ø)` | :arrow_down: | | [...t/common/response/broker/BrokerResponseNative.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVzcG9uc2UvYnJva2VyL0Jyb2tlclJlc3BvbnNlTmF0aXZlLmphdmE=) | `90.52% <0%> (+1.05%)` | `0% <0%> (ø)` | :arrow_down: | | [...e/io/writer/impl/MutableOffHeapByteArrayStore.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pby93cml0ZXIvaW1wbC9NdXRhYmxlT2ZmSGVhcEJ5dGVBcnJheVN0b3JlLmphdmE=) | `85.71% <0%> (+1.09%)` | `0% <0%> (ø)` | :arrow_down: | | [.../pinot/broker/broker/helix/HelixBrokerStarter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0hlbGl4QnJva2VyU3RhcnRlci5qYXZh) | `71.97% <0%> (+1.27%)` | `0% <0%> (ø)` | :arrow_down: | | ... and [303 more](https://codecov.io/gh/apache/incubator-pinot/pull/5148/diff?src=pr=tree-more) | | -- [Continue to review full report at
[GitHub] [incubator-pinot] codecov-io edited a comment on issue #5147: Support default star-tree
codecov-io edited a comment on issue #5147: Support default star-tree URL: https://github.com/apache/incubator-pinot/pull/5147#issuecomment-598421128 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=h1) Report > Merging [#5147](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/6b6fa56ab75e5f59f58249f7a733a375421b94b6=desc) will **increase** coverage by `8.98%`. > The diff coverage is `70.17%`. [](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#5147 +/- ## + Coverage 56.50% 65.49% +8.98% Complexity 12 12 Files 1052 1052 Lines 5413654190 +54 Branches 8080 8096 +16 + Hits 3059035492+4902 + Misses2110316077-5026 - Partials 2443 2621 +178 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/pinot/common/config/StarTreeIndexConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1N0YXJUcmVlSW5kZXhDb25maWcuamF2YQ==) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | | | [...main/java/org/apache/pinot/spi/data/FieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9GaWVsZFNwZWMuamF2YQ==) | `85.27% <0.00%> (-0.67%)` | `0.00 <0.00> (ø)` | | | [...indexsegment/generator/SegmentGeneratorConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pbmRleHNlZ21lbnQvZ2VuZXJhdG9yL1NlZ21lbnRHZW5lcmF0b3JDb25maWcuamF2YQ==) | `55.29% <40.00%> (+3.10%)` | `0.00 <0.00> (ø)` | | | [...core/startree/v2/builder/MultipleTreesBuilder.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS92Mi9idWlsZGVyL011bHRpcGxlVHJlZXNCdWlsZGVyLmphdmE=) | `82.22% <44.44%> (-9.67%)` | `0.00 <0.00> (ø)` | | | [...t/creator/impl/SegmentIndexCreationDriverImpl.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50SW5kZXhDcmVhdGlvbkRyaXZlckltcGwuamF2YQ==) | `81.81% <66.66%> (+1.24%)` | `0.00 <0.00> (ø)` | | | [...e/startree/v2/builder/StarTreeV2BuilderConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS92Mi9idWlsZGVyL1N0YXJUcmVlVjJCdWlsZGVyQ29uZmlnLmphdmE=) | `58.02% <81.81%> (+16.35%)` | `0.00 <0.00> (ø)` | | | [...org/apache/pinot/common/config/IndexingConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL0luZGV4aW5nQ29uZmlnLmphdmE=) | `100.00% <100.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...pql/parsers/PinotQuery2BrokerRequestConverter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9wcWwvcGFyc2Vycy9QaW5vdFF1ZXJ5MkJyb2tlclJlcXVlc3RDb252ZXJ0ZXIuamF2YQ==) | `88.07% <0.00%> (+0.45%)` | `0.00% <0.00%> (ø%)` | | | [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `88.25% <0.00%> (+0.75%)` | `0.00% <0.00%> (ø%)` | | | [.../helix/core/realtime/SegmentCompletionManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1NlZ21lbnRDb21wbGV0aW9uTWFuYWdlci5qYXZh) | `70.63% <0.00%> (+0.85%)` | `0.00% <0.00%> (ø%)` | | | ... and [308 more](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=continue). > **Legend** - [Click here to learn
[GitHub] [incubator-pinot] codecov-io edited a comment on issue #5147: Support default star-tree
codecov-io edited a comment on issue #5147: Support default star-tree URL: https://github.com/apache/incubator-pinot/pull/5147#issuecomment-598421128 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=h1) Report > Merging [#5147](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/6b6fa56ab75e5f59f58249f7a733a375421b94b6?src=pr=desc) will **decrease** coverage by `12.19%`. > The diff coverage is `14.75%`. [](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#5147 +/- ## - Coverage 56.5% 44.31% -12.2% Files 1052 1052 Lines 5413654184 +48 Branches 8080 8095 +15 - Hits 3059024012-6578 - Misses2110328089+6986 + Partials 2443 2083 -360 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/pinot/common/config/StarTreeIndexConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL1N0YXJUcmVlSW5kZXhDb25maWcuamF2YQ==) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...e/startree/v2/builder/StarTreeV2BuilderConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS92Mi9idWlsZGVyL1N0YXJUcmVlVjJCdWlsZGVyQ29uZmlnLmphdmE=) | `24.69% <0%> (-16.98%)` | `0 <0> (ø)` | | | [...main/java/org/apache/pinot/spi/data/FieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9GaWVsZFNwZWMuamF2YQ==) | `0% <0%> (-85.94%)` | `0 <0> (ø)` | | | [...org/apache/pinot/common/config/IndexingConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL0luZGV4aW5nQ29uZmlnLmphdmE=) | `100% <100%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...indexsegment/generator/SegmentGeneratorConfig.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9pbmRleHNlZ21lbnQvZ2VuZXJhdG9yL1NlZ21lbnRHZW5lcmF0b3JDb25maWcuamF2YQ==) | `46.2% <15.38%> (-5.99%)` | `0 <0> (ø)` | | | [...core/startree/v2/builder/MultipleTreesBuilder.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS92Mi9idWlsZGVyL011bHRpcGxlVHJlZXNCdWlsZGVyLmphdmE=) | `80% <25%> (-11.9%)` | `0 <0> (ø)` | | | [...t/creator/impl/SegmentIndexCreationDriverImpl.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50SW5kZXhDcmVhdGlvbkRyaXZlckltcGwuamF2YQ==) | `77.3% <42.85%> (-3.28%)` | `0 <0> (ø)` | | | [.../org/apache/pinot/common/http/MultiGetRequest.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vaHR0cC9NdWx0aUdldFJlcXVlc3QuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (ø)` | | | [...che/pinot/common/restlet/resources/TablesList.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVzdGxldC9yZXNvdXJjZXMvVGFibGVzTGlzdC5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (ø)` | | | [...ransformer/timeunit/CustomTimeUnitTransformer.java](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vdHJhbnNmb3JtZXIvdGltZXVuaXQvQ3VzdG9tVGltZVVuaXRUcmFuc2Zvcm1lci5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (ø)` | | | ... and [722 more](https://codecov.io/gh/apache/incubator-pinot/pull/5147/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5147?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5147: Support default star-tree
Jackie-Jiang commented on a change in pull request #5147: Support default
star-tree
URL: https://github.com/apache/incubator-pinot/pull/5147#discussion_r391966122
##
File path:
pinot-common/src/main/java/org/apache/pinot/common/config/IndexingConfig.java
##
@@ -145,6 +146,14 @@ public void setOnHeapDictionaryColumns(List
onHeapDictionaryColumns) {
_onHeapDictionaryColumns = onHeapDictionaryColumns;
}
+ public boolean isEnableDefaultStarTree() {
+return _enableDefaultStarTree;
+ }
+
+ public void setEnableDefaultStarTree(boolean enableDefaultStarTree) {
+_enableDefaultStarTree = enableDefaultStarTree;
Review comment:
I prefer the current way so that the default star-tree can be enabled by one
boolean flag. Don't want to change the existing StarTreeIndexConfig because
adding a mode seems more confusing to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5147: Support default star-tree
mcvsubbu commented on a change in pull request #5147: Support default star-tree
URL: https://github.com/apache/incubator-pinot/pull/5147#discussion_r391956217
##
File path:
pinot-common/src/main/java/org/apache/pinot/common/config/IndexingConfig.java
##
@@ -145,6 +146,14 @@ public void setOnHeapDictionaryColumns(List
onHeapDictionaryColumns) {
_onHeapDictionaryColumns = onHeapDictionaryColumns;
}
+ public boolean isEnableDefaultStarTree() {
+return _enableDefaultStarTree;
+ }
+
+ public void setEnableDefaultStarTree(boolean enableDefaultStarTree) {
+_enableDefaultStarTree = enableDefaultStarTree;
Review comment:
Another way to add this config may be to add a "mode" field in
StarTreeIndexConfig. If mode is set to "auto" then ignore other settings.
Otherwise, follow the other serttings.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5147: Support default star-tree
Jackie-Jiang commented on issue #5147: Support default star-tree URL: https://github.com/apache/incubator-pinot/pull/5147#issuecomment-598462189 > Also, please be sure to add documentation Documentation for default star-tree config is inside StarTreeV2BuilderConfig. Will add more documentation in gitbook later This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5147: Support default star-tree
Jackie-Jiang commented on a change in pull request #5147: Support default
star-tree
URL: https://github.com/apache/incubator-pinot/pull/5147#discussion_r391939779
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
##
@@ -190,13 +190,17 @@ public SegmentGeneratorConfig(@Nullable TableConfig
tableConfig, Schema schema)
_segmentPartitionConfig = indexingConfig.getSegmentPartitionConfig();
// Star-tree V2 configs
-List starTreeIndexConfigs =
indexingConfig.getStarTreeIndexConfigs();
-if (starTreeIndexConfigs != null && !starTreeIndexConfigs.isEmpty()) {
- List starTreeV2BuilderConfigs = new
ArrayList<>(starTreeIndexConfigs.size());
- for (StarTreeIndexConfig starTreeIndexConfig : starTreeIndexConfigs) {
-
starTreeV2BuilderConfigs.add(StarTreeV2BuilderConfig.fromIndexConfig(starTreeIndexConfig));
+if (indexingConfig.isEnableDefaultStarTree()) {
Review comment:
This won't have backward compatible issue because default star-tree is not
enabled by default.
But this is a great question, and we should not ignore the explicitly
configured star-tree. Modify the code to generate star-tree for both configs
(if both are defined, generate both default and customized).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5147: Support default star-tree
mcvsubbu commented on issue #5147: Support default star-tree URL: https://github.com/apache/incubator-pinot/pull/5147#issuecomment-598432625 Also, please be sure to add documentation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5147: Support default star-tree
mcvsubbu commented on a change in pull request #5147: Support default star-tree
URL: https://github.com/apache/incubator-pinot/pull/5147#discussion_r391916643
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
##
@@ -190,13 +190,17 @@ public SegmentGeneratorConfig(@Nullable TableConfig
tableConfig, Schema schema)
_segmentPartitionConfig = indexingConfig.getSegmentPartitionConfig();
// Star-tree V2 configs
-List starTreeIndexConfigs =
indexingConfig.getStarTreeIndexConfigs();
-if (starTreeIndexConfigs != null && !starTreeIndexConfigs.isEmpty()) {
- List starTreeV2BuilderConfigs = new
ArrayList<>(starTreeIndexConfigs.size());
- for (StarTreeIndexConfig starTreeIndexConfig : starTreeIndexConfigs) {
-
starTreeV2BuilderConfigs.add(StarTreeV2BuilderConfig.fromIndexConfig(starTreeIndexConfig));
+if (indexingConfig.isEnableDefaultStarTree()) {
Review comment:
If both are defined, you should pay attention to the specific start tree
definition requested. This will keep backward compat also.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
snleee commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r391915420
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(AzureGen2PinotFS.class);
+
+ private static final String ACCOUNT_NAME = "accountName";
+ private static final String ACCESS_KEY = "accessKey";
+ private static final String FILE_SYSTEM_NAME = "fileSystemName";
+ private static final String ENABLE_CHECKSUM = "enableChecksum";
+
+ private static final String HTTPS_URL_PREFIX = "https://;;
+
+ private static final String AZURE_STORAGE_DNS_SUFFIX =
".dfs.core.windows.net";
+ private static final String AZURE_BLOB_DNS_SUFFIX = ".blob.core.windows.net";
+ private static final String PATH_ALREADY_EXISTS_ERROR_CODE =
"PathAlreadyExists";
+ private static final String IS_DIRECTORY_KEY = "hdi_isfolder";
+
+ private static final int NOT_FOUND_STATUS_CODE = 404;
+ private static final int ALREADY_EXISTS_STATUS_CODE = 409;
+
+ // Azure Data Lake Gen2's block size is 4MB
+ private static final int BUFFER_SIZE = 4 * 1024 * 1024;
+
+ private DataLakeFileSystemClient _fileSystemClient;
+ private BlobServiceClient _blobServiceClient;
+
+ // If enabled, pinotFS implementation will guarantee that the bits you've
read are the same as the ones you wrote.
+ // However, there's some overhead in computing hash. (Adds roughly 3 seconds
for 1GB file)
+ private boolean _enableChecksum;
+
+ @Override
+ public void init(Configuration config) {
+_enableChecksum = config.getBoolean(ENABLE_CHECKSUM, false);
+
+// Azure storage account name
+String accountName = config.getString(ACCOUNT_NAME);
+
+// TODO: consider to add the encryption of the following config
+String accessKey = config.getString(ACCESS_KEY);
+String fileSystemName = config.getString(FILE_SYSTEM_NAME);
+
+String dfsServiceEndpointUrl = HTTPS_URL_PREFIX + accountName +
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r391853187
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(AzureGen2PinotFS.class);
+
+ private static final String ACCOUNT_NAME = "accountName";
+ private static final String ACCESS_KEY = "accessKey";
+ private static final String FILE_SYSTEM_NAME = "fileSystemName";
+ private static final String ENABLE_CHECKSUM = "enableChecksum";
+
+ private static final String HTTPS_URL_PREFIX = "https://;;
+
+ private static final String AZURE_STORAGE_DNS_SUFFIX =
".dfs.core.windows.net";
+ private static final String AZURE_BLOB_DNS_SUFFIX = ".blob.core.windows.net";
+ private static final String PATH_ALREADY_EXISTS_ERROR_CODE =
"PathAlreadyExists";
+ private static final String IS_DIRECTORY_KEY = "hdi_isfolder";
+
+ private static final int NOT_FOUND_STATUS_CODE = 404;
+ private static final int ALREADY_EXISTS_STATUS_CODE = 409;
+
+ // Azure Data Lake Gen2's block size is 4MB
+ private static final int BUFFER_SIZE = 4 * 1024 * 1024;
+
+ private DataLakeFileSystemClient _fileSystemClient;
+ private BlobServiceClient _blobServiceClient;
+
+ // If enabled, pinotFS implementation will guarantee that the bits you've
read are the same as the ones you wrote.
+ // However, there's some overhead in computing hash. (Adds roughly 3 seconds
for 1GB file)
+ private boolean _enableChecksum;
+
+ @Override
+ public void init(Configuration config) {
+_enableChecksum = config.getBoolean(ENABLE_CHECKSUM, false);
+
+// Azure storage account name
+String accountName = config.getString(ACCOUNT_NAME);
+
+// TODO: consider to add the encryption of the following config
+String accessKey = config.getString(ACCESS_KEY);
+String fileSystemName = config.getString(FILE_SYSTEM_NAME);
+
+String dfsServiceEndpointUrl = HTTPS_URL_PREFIX + accountName +
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r391856425
##
File path: pinot-spi/src/main/java/org/apache/pinot/spi/filesystem/PinotFS.java
##
@@ -103,8 +106,13 @@ public boolean move(URI srcUri, URI dstUri, boolean
overwrite)
}
} else {
// ensures the parent path of dst exists.
- URI parentUri = Paths.get(dstUri).getParent().toUri();
- mkdir(parentUri);
+ try {
+Path parentPath = Paths.get(dstUri.getPath()).getParent();
Review comment:
Can you add a test for testing the case when the scheme doesn't match then
throw the exception?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r391857536
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
Review comment:
Do we have any test for this class? We've noticed the code coverage is only
50% though.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
jackjlli commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r391851562
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(AzureGen2PinotFS.class);
+
+ private static final String ACCOUNT_NAME = "accountName";
+ private static final String ACCESS_KEY = "accessKey";
+ private static final String FILE_SYSTEM_NAME = "fileSystemName";
+ private static final String ENABLE_CHECKSUM = "enableChecksum";
+
+ private static final String HTTPS_URL_PREFIX = "https://;;
+
+ private static final String AZURE_STORAGE_DNS_SUFFIX =
".dfs.core.windows.net";
+ private static final String AZURE_BLOB_DNS_SUFFIX = ".blob.core.windows.net";
+ private static final String PATH_ALREADY_EXISTS_ERROR_CODE =
"PathAlreadyExists";
+ private static final String IS_DIRECTORY_KEY = "hdi_isfolder";
+
+ private static final int NOT_FOUND_STATUS_CODE = 404;
+ private static final int ALREADY_EXISTS_STATUS_CODE = 409;
+
+ // Azure Data Lake Gen2's block size is 4MB
+ private static final int BUFFER_SIZE = 4 * 1024 * 1024;
+
+ private DataLakeFileSystemClient _fileSystemClient;
+ private BlobServiceClient _blobServiceClient;
+
+ // If enabled, pinotFS implementation will guarantee that the bits you've
read are the same as the ones you wrote.
+ // However, there's some overhead in computing hash. (Adds roughly 3 seconds
for 1GB file)
+ private boolean _enableChecksum;
+
+ @Override
+ public void init(Configuration config) {
+_enableChecksum = config.getBoolean(ENABLE_CHECKSUM, false);
+
+// Azure storage account name
+String accountName = config.getString(ACCOUNT_NAME);
+
+// TODO: consider to add the encryption of the following config
+String accessKey = config.getString(ACCESS_KEY);
+String fileSystemName = config.getString(FILE_SYSTEM_NAME);
+
+String dfsServiceEndpointUrl = HTTPS_URL_PREFIX + accountName +
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5146: Update pinot assembly scripts for release 0.3.0
fx19880617 commented on a change in pull request #5146: Update pinot assembly
scripts for release 0.3.0
URL: https://github.com/apache/incubator-pinot/pull/5146#discussion_r391854918
##
File path: pinot-distribution/pinot-assembly.xml
##
@@ -43,10 +43,74 @@
${pinot.root}/DISCLAIMER
+
${pinot.root}/pinot-distribution/target/pinot-distribution-${project.version}-shaded.jar
lib/pinot-all-${project.version}-jar-with-dependencies.jar
+
+
+
+
+
${pinot.root}/pinot-plugins/pinot-stream-ingestion/pinot-kafka-${kafka.version}/target/pinot-kafka-${kafka.version}-${project.version}-shaded.jar
Review comment:
That is how we did it previously by gathering all the jars together then
copy directory, see line
`${pinot.root}/pinot-plugins/target/plugins/${plugin.type}/${project.artifactId}`
in `pinot-plugins/pom.xml`.
However `mvn release` doesn't honor this, and cannot find the built jars if
we relocated them, that's why we need to hand-pick all those jars again here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] snleee commented on a change in pull request #5146: Update pinot assembly scripts for release 0.3.0
snleee commented on a change in pull request #5146: Update pinot assembly
scripts for release 0.3.0
URL: https://github.com/apache/incubator-pinot/pull/5146#discussion_r391849492
##
File path: pinot-distribution/pinot-assembly.xml
##
@@ -43,10 +43,74 @@
${pinot.root}/DISCLAIMER
+
${pinot.root}/pinot-distribution/target/pinot-distribution-${project.version}-shaded.jar
lib/pinot-all-${project.version}-jar-with-dependencies.jar
+
+
+
+
+
${pinot.root}/pinot-plugins/pinot-stream-ingestion/pinot-kafka-${kafka.version}/target/pinot-kafka-${kafka.version}-${project.version}-shaded.jar
Review comment:
After I run `mvn install`, I see `pinot-plugins/target/plugins` has all
shaded plugins copied. Is it possible to copy this directory at once instead of
listing all files?
```
~/workspace/pinot/pinot-plugins/target/plugins
❯ pwd
/Users/snlee/workspace/pinot/pinot-plugins/target/plugins
~/workspace/pinot/pinot-plugins/target/plugins
❯ ll
total 0
drwxr-xr-x 11 snlee LINKEDIN\eng 352B Mar 10 21:33 pinot-batch-ingestion
drwxr-xr-x 6 snlee LINKEDIN\eng 192B Mar 10 21:30 pinot-file-system
drwxr-xr-x 9 snlee LINKEDIN\eng 288B Mar 10 21:30 pinot-input-format
drwxr-xr-x 2 snlee LINKEDIN\eng64B Mar 10 21:29 pinot-plugins
drwxr-xr-x 6 snlee LINKEDIN\eng 192B Mar 10 21:33 pinot-stream-ingestion
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] snleee commented on a change in pull request #5146: Update pinot assembly scripts for release 0.3.0
snleee commented on a change in pull request #5146: Update pinot assembly
scripts for release 0.3.0
URL: https://github.com/apache/incubator-pinot/pull/5146#discussion_r391849492
##
File path: pinot-distribution/pinot-assembly.xml
##
@@ -43,10 +43,74 @@
${pinot.root}/DISCLAIMER
+
${pinot.root}/pinot-distribution/target/pinot-distribution-${project.version}-shaded.jar
lib/pinot-all-${project.version}-jar-with-dependencies.jar
+
+
+
+
+
${pinot.root}/pinot-plugins/pinot-stream-ingestion/pinot-kafka-${kafka.version}/target/pinot-kafka-${kafka.version}-${project.version}-shaded.jar
Review comment:
After I run `mvn install`, I see `pinot-plugins/target/plugins` has all
shaded plugins copied. Is it possible to copy this directory at once instead of
listing all files?
```
~/workspace/pinot/pinot-plugins/target/plugins adls-gen2*
❯ pwd
/Users/snlee/workspace/pinot/pinot-plugins/target/plugins
~/workspace/pinot/pinot-plugins/target/plugins adls-gen2*
❯ ll
total 0
drwxr-xr-x 11 snlee LINKEDIN\eng 352B Mar 10 21:33 pinot-batch-ingestion
drwxr-xr-x 6 snlee LINKEDIN\eng 192B Mar 10 21:30 pinot-file-system
drwxr-xr-x 9 snlee LINKEDIN\eng 288B Mar 10 21:30 pinot-input-format
drwxr-xr-x 2 snlee LINKEDIN\eng64B Mar 10 21:29 pinot-plugins
drwxr-xr-x 6 snlee LINKEDIN\eng 192B Mar 10 21:33 pinot-stream-ingestion
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5147: Support default star-tree
Jackie-Jiang opened a new pull request #5147: Support default star-tree URL: https://github.com/apache/incubator-pinot/pull/5147 Support generating default star-tree config with the following rules: - All dictionary-encoded single-value dimensions (including date-time columns) with cardinality smaller or equal to the threshold (1) will be included in the split order, sorted by their cardinality in descending order - Time column (if exists and dictionary-encoded) will be appended to the split order as the last element - Use COUNT(*) and SUM for all numeric metrics as function column pairs - Use default value (1) for max leaf records This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (f7f9f4a -> 6b6fa56a)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from f7f9f4a Fix hybrid quickstart (#5143)
add 6b6fa56a Update license and notice (#5145)
No new revisions were added by this update.
Summary of changes:
LICENSE-binary | 131 +-
NOTICE | 2 +-
NOTICE-binary | 448 +
...LICENSE-protobuf.txt => LICENSE-fusesource.txt} | 13 +-
licenses-binary/LICENSE-py4j.txt | 26 ++
licenses-binary/LICENSE-threeten.txt | 29 ++
6 files changed, 452 insertions(+), 197 deletions(-)
copy licenses-binary/{LICENSE-protobuf.txt => LICENSE-fusesource.txt} (71%)
create mode 100644 licenses-binary/LICENSE-py4j.txt
create mode 100644 licenses-binary/LICENSE-threeten.txt
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 merged pull request #5145: Update license and notice for 0.3.0 release
fx19880617 merged pull request #5145: Update license and notice for 0.3.0 release URL: https://github.com/apache/incubator-pinot/pull/5145 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5116: Add Azure Data Lake Gen2 connector for PinotFS
mcvsubbu commented on a change in pull request #5116: Add Azure Data Lake Gen2
connector for PinotFS
URL: https://github.com/apache/incubator-pinot/pull/5116#discussion_r391792039
##
File path:
pinot-plugins/pinot-file-system/pinot-adls/src/main/java/org/apache/pinot/plugin/filesystem/AzureGen2PinotFS.java
##
@@ -0,0 +1,460 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.plugin.filesystem;
+
+import com.azure.core.http.rest.PagedIterable;
+import com.azure.core.util.Context;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.common.StorageSharedKeyCredential;
+import com.azure.storage.common.Utility;
+import com.azure.storage.file.datalake.DataLakeDirectoryClient;
+import com.azure.storage.file.datalake.DataLakeFileClient;
+import com.azure.storage.file.datalake.DataLakeFileSystemClient;
+import com.azure.storage.file.datalake.DataLakeServiceClient;
+import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
+import com.azure.storage.file.datalake.models.DataLakeRequestConditions;
+import com.azure.storage.file.datalake.models.DataLakeStorageException;
+import com.azure.storage.file.datalake.models.ListPathsOptions;
+import com.azure.storage.file.datalake.models.PathHttpHeaders;
+import com.azure.storage.file.datalake.models.PathItem;
+import com.azure.storage.file.datalake.models.PathProperties;
+import java.io.ByteArrayInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.sql.Timestamp;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.Map;
+import org.apache.commons.configuration.Configuration;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.spi.filesystem.PinotFS;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Azure Data Lake Storage Gen2 implementation for the PinotFS interface.
+ */
+public class AzureGen2PinotFS extends PinotFS {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(AzureGen2PinotFS.class);
+
+ private static final String ACCOUNT_NAME = "accountName";
+ private static final String ACCESS_KEY = "accessKey";
+ private static final String FILE_SYSTEM_NAME = "fileSystemName";
+ private static final String ENABLE_CHECKSUM = "enableChecksum";
+
+ private static final String HTTPS_URL_PREFIX = "https://;;
+
+ private static final String AZURE_STORAGE_DNS_SUFFIX =
".dfs.core.windows.net";
+ private static final String AZURE_BLOB_DNS_SUFFIX = ".blob.core.windows.net";
+ private static final String PATH_ALREADY_EXISTS_ERROR_CODE =
"PathAlreadyExists";
+ private static final String IS_DIRECTORY_KEY = "hdi_isfolder";
+
+ private static final int NOT_FOUND_STATUS_CODE = 404;
+ private static final int ALREADY_EXISTS_STATUS_CODE = 409;
+
+ // Azure Data Lake Gen2's block size is 4MB
+ private static final int BUFFER_SIZE = 4 * 1024 * 1024;
+
+ private DataLakeFileSystemClient _fileSystemClient;
+ private BlobServiceClient _blobServiceClient;
+
+ // If enabled, pinotFS implementation will guarantee that the bits you've
read are the same as the ones you wrote.
+ // However, there's some overhead in computing hash. (Adds roughly 3 seconds
for 1GB file)
+ private boolean _enableChecksum;
+
+ @Override
+ public void init(Configuration config) {
+_enableChecksum = config.getBoolean(ENABLE_CHECKSUM, false);
+
+// Azure storage account name
+String accountName = config.getString(ACCOUNT_NAME);
+
+// TODO: consider to add the encryption of the following config
+String accessKey = config.getString(ACCESS_KEY);
+String fileSystemName = config.getString(FILE_SYSTEM_NAME);
+
+String dfsServiceEndpointUrl = HTTPS_URL_PREFIX + accountName +
[GitHub] [incubator-pinot] codecov-io edited a comment on issue #5145: Update license and notice for 0.3.0 release
codecov-io edited a comment on issue #5145: Update license and notice for 0.3.0 release URL: https://github.com/apache/incubator-pinot/pull/5145#issuecomment-598070125 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5145?src=pr=h1) Report > Merging [#5145](https://codecov.io/gh/apache/incubator-pinot/pull/5145?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/f7f9f4aa5fb6e87308a155a2adfc06c7fa7fcfab=desc) will **increase** coverage by `0.08%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-pinot/pull/5145?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#5145 +/- ## + Coverage 65.55% 65.64% +0.08% Complexity 12 12 Files 1052 1052 Lines 5413654136 Branches 8080 8080 + Hits 3548935536 +47 + Misses1602915960 -69 - Partials 2618 2640 +22 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5145?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/impl/dictionary/LongOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09uSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=) | `64.63% <0.00%> (-12.20%)` | `0.00% <0.00%> (ø%)` | | | [...ore/query/scheduler/resources/ResourceManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvcmVzb3VyY2VzL1Jlc291cmNlTWFuYWdlci5qYXZh) | `85.71% <0.00%> (-10.72%)` | `0.00% <0.00%> (ø%)` | | | [...elix/core/relocation/RealtimeSegmentRelocator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlbG9jYXRpb24vUmVhbHRpbWVTZWdtZW50UmVsb2NhdG9yLmphdmE=) | `34.21% <0.00%> (-10.53%)` | `0.00% <0.00%> (ø%)` | | | [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `92.30% <0.00%> (-7.70%)` | `0.00% <0.00%> (ø%)` | | | [...impl/dictionary/DoubleOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `47.56% <0.00%> (-7.32%)` | `0.00% <0.00%> (ø%)` | | | [.../pinot/core/query/scheduler/PriorityScheduler.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvUHJpb3JpdHlTY2hlZHVsZXIuamF2YQ==) | `78.08% <0.00%> (-5.48%)` | `0.00% <0.00%> (ø%)` | | | [...e/operator/dociditerators/MVScanDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9NVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `60.31% <0.00%> (-3.18%)` | `0.00% <0.00%> (ø%)` | | | [...e/operator/dociditerators/SortedDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9Tb3J0ZWREb2NJZEl0ZXJhdG9yLmphdmE=) | `61.11% <0.00%> (-2.78%)` | `0.00% <0.00%> (ø%)` | | | [...e/operator/dociditerators/SVScanDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9TVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `63.21% <0.00%> (-2.30%)` | `0.00% <0.00%> (ø%)` | | | [...g/apache/pinot/common/utils/helix/HelixHelper.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaGVsaXgvSGVsaXhIZWxwZXIuamF2YQ==) | `51.49% <0.00%> (-1.80%)` | `0.00% <0.00%> (ø%)` | | | ... and [25 more](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree-more) | | -- [Continue to review full report at
[GitHub] [incubator-pinot] codecov-io commented on issue #5145: Update license and notice for 0.3.0 release
codecov-io commented on issue #5145: Update license and notice for 0.3.0 release URL: https://github.com/apache/incubator-pinot/pull/5145#issuecomment-598070125 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5145?src=pr=h1) Report > Merging [#5145](https://codecov.io/gh/apache/incubator-pinot/pull/5145?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/f7f9f4aa5fb6e87308a155a2adfc06c7fa7fcfab=desc) will **increase** coverage by `0.08%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-pinot/pull/5145?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#5145 +/- ## + Coverage 65.55% 65.64% +0.08% Complexity 12 12 Files 1052 1052 Lines 5413654136 Branches 8080 8080 + Hits 3548935536 +47 + Misses1602915960 -69 - Partials 2618 2640 +22 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5145?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/impl/dictionary/LongOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvTG9uZ09uSGVhcE11dGFibGVEaWN0aW9uYXJ5LmphdmE=) | `64.63% <0.00%> (-12.20%)` | `0.00% <0.00%> (ø%)` | | | [...ore/query/scheduler/resources/ResourceManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvcmVzb3VyY2VzL1Jlc291cmNlTWFuYWdlci5qYXZh) | `85.71% <0.00%> (-10.72%)` | `0.00% <0.00%> (ø%)` | | | [...elix/core/relocation/RealtimeSegmentRelocator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlbG9jYXRpb24vUmVhbHRpbWVTZWdtZW50UmVsb2NhdG9yLmphdmE=) | `34.21% <0.00%> (-10.53%)` | `0.00% <0.00%> (ø%)` | | | [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `92.30% <0.00%> (-7.70%)` | `0.00% <0.00%> (ø%)` | | | [...impl/dictionary/DoubleOnHeapMutableDictionary.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yZWFsdGltZS9pbXBsL2RpY3Rpb25hcnkvRG91YmxlT25IZWFwTXV0YWJsZURpY3Rpb25hcnkuamF2YQ==) | `47.56% <0.00%> (-7.32%)` | `0.00% <0.00%> (ø%)` | | | [.../pinot/core/query/scheduler/PriorityScheduler.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvUHJpb3JpdHlTY2hlZHVsZXIuamF2YQ==) | `78.08% <0.00%> (-5.48%)` | `0.00% <0.00%> (ø%)` | | | [...e/operator/dociditerators/MVScanDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9NVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `60.31% <0.00%> (-3.18%)` | `0.00% <0.00%> (ø%)` | | | [...e/operator/dociditerators/SortedDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9Tb3J0ZWREb2NJZEl0ZXJhdG9yLmphdmE=) | `61.11% <0.00%> (-2.78%)` | `0.00% <0.00%> (ø%)` | | | [...e/operator/dociditerators/SVScanDocIdIterator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9TVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `63.21% <0.00%> (-2.30%)` | `0.00% <0.00%> (ø%)` | | | [...g/apache/pinot/common/utils/helix/HelixHelper.java](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaGVsaXgvSGVsaXhIZWxwZXIuamF2YQ==) | `51.49% <0.00%> (-1.80%)` | `0.00% <0.00%> (ø%)` | | | ... and [25 more](https://codecov.io/gh/apache/incubator-pinot/pull/5145/diff?src=pr=tree-more) | | -- [Continue to review full report at
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5146: update pinot assembly scripts
fx19880617 opened a new pull request #5146: update pinot assembly scripts URL: https://github.com/apache/incubator-pinot/pull/5146 This is for release 0.3.0 `mvn release` command doesn't honor output relocation, so we need to manually specify those in assembly scripts for `pinot-plugins`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch update_assembly_scripts created (now 7ed5ce3)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch update_assembly_scripts in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 7ed5ce3 update pinot assembly scripts This branch includes the following new commits: new 7ed5ce3 update pinot assembly scripts The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: update pinot assembly scripts
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch update_assembly_scripts
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 7ed5ce305898ab02981e1aba4ae9fa09628ace2c
Author: Xiang Fu
AuthorDate: Mon Mar 2 16:53:01 2020 -0800
update pinot assembly scripts
---
pinot-distribution/pinot-assembly.xml | 88 +--
pinot-plugins/pom.xml | 1 -
2 files changed, 64 insertions(+), 25 deletions(-)
diff --git a/pinot-distribution/pinot-assembly.xml
b/pinot-distribution/pinot-assembly.xml
index 1bc4051..c78889b 100644
--- a/pinot-distribution/pinot-assembly.xml
+++ b/pinot-distribution/pinot-assembly.xml
@@ -43,10 +43,74 @@
${pinot.root}/DISCLAIMER
+
${pinot.root}/pinot-distribution/target/pinot-distribution-${project.version}-shaded.jar
lib/pinot-all-${project.version}-jar-with-dependencies.jar
+
+
+
+
+
${pinot.root}/pinot-plugins/pinot-stream-ingestion/pinot-kafka-${kafka.version}/target/pinot-kafka-${kafka.version}-${project.version}-shaded.jar
+
plugins/pinot-stream-ingestion/pinot-kafka-${kafka.version}/pinot-kafka-${kafka.version}-${project.version}-shaded.jar
+
+
+
+
+
${pinot.root}/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/target/pinot-batch-ingestion-standalone-${project.version}-shaded.jar
+
plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/pinot-batch-ingestion-standalone-${project.version}-shaded.jar
+
+
+
${pinot.root}/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/target/pinot-batch-ingestion-hadoop-${project.version}-shaded.jar
+
plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/pinot-batch-ingestion-hadoop-${project.version}-shaded.jar
+
+
+
${pinot.root}/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/target/pinot-batch-ingestion-spark-${project.version}-shaded.jar
+
plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/pinot-batch-ingestion-spark-${project.version}-shaded.jar
+
+
+
+
+
${pinot.root}/pinot-plugins/pinot-file-system/pinot-adls/target/pinot-adls-${project.version}-shaded.jar
+
plugins/pinot-file-system/pinot-adls/pinot-adls-${project.version}-shaded.jar
+
+
+
${pinot.root}/pinot-plugins/pinot-file-system/pinot-gcs/target/pinot-gcs-${project.version}-shaded.jar
+
plugins/pinot-file-system/pinot-gcs/pinot-gcs-${project.version}-shaded.jar
+
+
+
${pinot.root}/pinot-plugins/pinot-file-system/pinot-hdfs/target/pinot-hdfs-${project.version}-shaded.jar
+
plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-${project.version}-shaded.jar
+
+
+
+
+
${pinot.root}/pinot-plugins/pinot-input-format/pinot-avro/target/pinot-avro-${project.version}-shaded.jar
+
plugins/pinot-input-format/pinot-avro/pinot-avro-${project.version}-shaded.jar
+
+
+
${pinot.root}/pinot-plugins/pinot-input-format/pinot-csv/target/pinot-csv-${project.version}-shaded.jar
+
plugins/pinot-input-format/pinot-csv/pinot-csv-${project.version}-shaded.jar
+
+
+
${pinot.root}/pinot-plugins/pinot-input-format/pinot-json/target/pinot-json-${project.version}-shaded.jar
+
plugins/pinot-input-format/pinot-json/pinot-json-${project.version}-shaded.jar
+
+
+
${pinot.root}/pinot-plugins/pinot-input-format/pinot-orc/target/pinot-orc-${project.version}-shaded.jar
+
plugins/pinot-input-format/pinot-orc/pinot-orc-${project.version}-shaded.jar
+
+
+
${pinot.root}/pinot-plugins/pinot-input-format/pinot-parquet/target/pinot-parquet-${project.version}-shaded.jar
+
plugins/pinot-input-format/pinot-parquet/pinot-parquet-${project.version}-shaded.jar
+
+
+
${pinot.root}/pinot-plugins/pinot-input-format/pinot-thrift/target/pinot-thrift-${project.version}-shaded.jar
+
plugins/pinot-input-format/pinot-thrift/pinot-thrift-${project.version}-shaded.jar
+
+
+
@@ -86,29 +150,5 @@
bin
0755
-
- false
- ${pinot.root}/pinot-plugins/target/plugins/
- plugins
-
-**/*.pom
-**/pinot-plugins/**
-**/pinot-file-system/pinot-file-system/**
-**/pinot-input-format/pinot-input-format/**
-**/pinot-stream-ingestion/pinot-stream-ingestion/**
-**/pinot-stream-ingestion/pinot-kafka-*/**
-**/pinot-batch-ingestion/pinot-batch-ingestion/**
-
**/pinot-batch-ingestion/pinot-batch-ingestion-common/**
-**/pinot-batch-ingestion/pinot-ingestion-common/**
-**/pinot-batch-ingestion/pinot-hadoop/**
-**/pinot-batch-ingestion/pinot-spark/**
-**/pinot-batch-ingestion/v0_deprecated/**
-
-
-
- false
-
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5145: Update license and notice for 0.3.0 release
fx19880617 opened a new pull request #5145: Update license and notice for 0.3.0 release URL: https://github.com/apache/incubator-pinot/pull/5145 Updated LICENSE and NOTICE. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch update_license_notice_0.3.0 created (now 31bca2e)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch update_license_notice_0.3.0 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 31bca2e Update license and notice This branch includes the following new commits: new 31bca2e Update license and notice The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Update license and notice
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a commit to branch update_license_notice_0.3.0 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 31bca2ea377feefc5acaa1dd931755e96b72db20 Author: Xiang Fu AuthorDate: Mon Mar 2 03:58:37 2020 -0800 Update license and notice --- LICENSE-binary | 131 +- NOTICE | 2 +- NOTICE-binary | 448 - licenses-binary/LICENSE-fusesource.txt | 27 ++ licenses-binary/LICENSE-py4j.txt | 26 ++ licenses-binary/LICENSE-threeten.txt | 29 +++ 6 files changed, 475 insertions(+), 188 deletions(-) diff --git a/LICENSE-binary b/LICENSE-binary index 49767f3..5c0e3e4 100644 --- a/LICENSE-binary +++ b/LICENSE-binary @@ -206,6 +206,7 @@ License Version 2.0: com.101tec:zkclient:0.7 com.amazonaws:aws-java-sdk:1.7.4 +com.beust.jcommander:1.64 com.clearspring.analytics:stream:2.7.0 com.fasterxml.jackson.core:jackson-annotations:2.9.8 com.fasterxml.jackson.core:jackson-core:2.9.8 @@ -214,40 +215,76 @@ com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.9.8 com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.9.8 com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.9.8 com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.9.8 +com.fasterxml.jackson.module:jackson-module-paranamer:2.6.5 +com.fasterxml.jackson.module:jackson-module-scala_2.11:2.6.5 +com.github.jnr:jffi:1.2.9 +com.github.jnr:jnr-constants:0.8.7 +com.github.jnr:jnr-ffi:2.0.3 +com.google.api.grpc:proto-google-common-protos:1.17.0 +com.google.api.grpc:proto-google-iam-v1:0.13.0 +com.google.api:api-common:1.8.1 +com.google.api:gax:1.49.1 +com.google.api:gax-httpjson:0.66.1 +com.google.api:google-api-client:1.30.4 +com.google.api:google-api-services-storage:v1-rev20190910-1.30.3.jar +com.google.auth:google-auth-library-credentials:0.18.0 +com.google.auth:google-auth-library-oauth2-http:0.18.0 +com.google.auto.value:auto-value-annotations:1.6.6 +com.google.cloud:google-cloud-core:1.91.3 +com.google.cloud:google-cloud-core-http:1.91.3 +com.google.cloud:google-cloud-storage:1.101.0 com.google.code.findbugs:jsr305:3.0.0 com.google.code.gson:gson:2.2.4 +com.google.errorprone:error_prone_annotations:2.3.2 com.google.guava:guava:20.0 +com.google.http-client:google-http-client-appengine:1.32.1 +com.google.http-client:google-http-client-jackson2:1.32.1 +com.google.http-client:google-http-client:1.32.1 com.google.inject.extensions:guice-servlet:3.0 com.google.inject:guice:3.0 +com.google.j2objc:j2objc-annotations:1.3 +com.google.oauth-client:google-oauth-client:1.30.3 +com.google.protobuf:protobuf-java-util:3.10.0 +com.google.protobuf:protobuf-java:2.5.0 +com.google.protobuf:protobuf-java:3.10.0 com.jamesmurty.utils:java-xmlbuilder:0.4 com.lmax:disruptor:3.3.4 +com.ning:async-http-client:1.9.21 +com.ning:compress-lzf:1.0.3 +com.tdunning:t-digest:3.2 +com.twitter:chill-java:0.8.0 +com.twitter:chill_2.11:0.8.0 +com.typesafe:config:1.3.2 +com.typesafe.scala-logging:scala-logging_2.11:3.9.0 +com.yammer.metrics:metrics-core:2.2.0 commons-beanutils:commons-beanutils:1.8.3 commons-beanutils:commons-beanutils-core:1.8.0 commons-cli:commons-cli:1.2 commons-codec:commons-codec:1.6 commons-collections:commons-collections:3.2.1 commons-configuration:commons-configuration:1.6 +commons-daemon:commons-daemon:1.0.13 commons-digester:commons-digester:1.8 commons-httpclient:commons-httpclient:3.1 commons-io:commons-io:2.4 commons-lang:commons-lang:2.6 commons-logging:commons-logging:1.2 commons-net:commons-net:3.1 -com.ning:async-http-client:1.9.21 -com.tdunning:t-digest:3.2 -com.typesafe:config:1.3.2 -com.yammer.metrics:metrics-core:2.2.0 io.airlift:aircompressor:0.10 io.dropwizard.metrics:metrics-core:3.2.3 +io.dropwizard.metrics:metrics-graphite:3.1.2 +io.dropwizard.metrics:metrics-json:3.1.2 +io.dropwizard.metrics:metrics-jvm:3.1.2 +io.grpc:grpc-context:1.22.1 io.netty:netty:3.9.6.Final io.netty:netty-all:4.1.28.Final +io.opencensus:opencensus-api:0.24.0 +io.opencensus:opencensus-contrib-http-util:0.24.0 io.swagger:swagger-annotations:1.5.16 io.swagger:swagger-core:1.5.16 io.swagger:swagger-jaxrs:1.5.16 io.swagger:swagger-jersey2-jaxrs:1.5.16 io.swagger:swagger-models:1.5.16 -io.vavr:vavr:0.9.2 -io.vavr:vavr-match:0.9.2 it.unimi.dsi:fastutil:8.2.3 javax.inject:javax.inject:1 javax.servlet.jsp:jsp-api:2.1 @@ -255,20 +292,25 @@ javax.servlet:servlet-api:2.5 javax.validation:validation-api:2.0.1.Final joda-time:joda-time:2.0 me.lemire.integercompression:JavaFastPFOR:0.0.13 -net.java.dev.jets3t:jets3t:0.9.0 +mx4j:mx4j:3.0.2 +net.bytebuddy:byte-buddy-agent:1.7.4 +net.bytebuddy:byte-buddy:1.7.4 +net.java.dev.jets3t:jets3t:0.9.3 net.jpountz.lz4:lz4:1.2.0 +nl.jqno.equalsverifier:equalsverifier:1.7.2 org.apache.avro:avro:1.7.6 org.apache.avro:avro-ipc:1.7.4 org.apache.avro:avro-mapred:1.7.4
