[GitHub] [pinot] navina opened a new pull request, #9851: Merge new columns in existing record with default merge strategy

2022-11-22 Thread GitBox
navina opened a new pull request, #9851: URL: https://github.com/apache/pinot/pull/9851 Related to #9771 After a schema evolves (eg. new column added), the column is not treated as an upsert column by the partition upsert manager because it is not part of

[GitHub] [pinot] codecov-commenter commented on pull request #9849: Combine the read access for replication config

2022-11-22 Thread GitBox
codecov-commenter commented on PR #9849: URL: https://github.com/apache/pinot/pull/9849#issuecomment-1324630346 # [Codecov](https://codecov.io/gh/apache/pinot/pull/9849?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report

[GitHub] [pinot] codecov-commenter commented on pull request #9850: [multistage][testing] Basic math func test and some more between test

2022-11-22 Thread GitBox
codecov-commenter commented on PR #9850: URL: https://github.com/apache/pinot/pull/9850#issuecomment-1324626639 # [Codecov](https://codecov.io/gh/apache/pinot/pull/9850?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report

[GitHub] [pinot] 61yao opened a new pull request, #9850: [multistage][testing] Basic math func test and some more between test

2022-11-22 Thread GitBox
61yao opened a new pull request, #9850: URL: https://github.com/apache/pinot/pull/9850 1) Add basic math func test 2) Add between literal test 3) Add between test for string type -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [pinot] snleee commented on pull request #9849: Combine the read access for replication config

2022-11-22 Thread GitBox
snleee commented on PR #9849: URL: https://github.com/apache/pinot/pull/9849#issuecomment-1324586307 https://github.com/apache/pinot/issues/8804 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [pinot] snleee opened a new pull request, #9849: Combine the read access for replication config

2022-11-22 Thread GitBox
snleee opened a new pull request, #9849: URL: https://github.com/apache/pinot/pull/9849 Currently, we have a separate configuration for replication. Offline and HLC reads from `replication` and LLC reads from `replicasPerPartition`. This PR combines the read access for the replication

[GitHub] [pinot] yupeng9 commented on issue #8804: Replication vs ReplicasPerPartition

2022-11-22 Thread GitBox
yupeng9 commented on issue #8804: URL: https://github.com/apache/pinot/issues/8804#issuecomment-1324529479 Missed this thread. Uber uses LLC only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [pinot] snleee opened a new issue, #9847: Add gz file support for `ORCRecordReader` and `ParquetAvroRecordReader`

2022-11-22 Thread GitBox
snleee opened a new issue, #9847: URL: https://github.com/apache/pinot/issues/9847 Currently, many record reader implementation supports to read either raw file or gzip file as an input. For instance, csv record reader can understand both `data.csv` and `data.csv.gz`. Here are the

[GitHub] [pinot] 61yao commented on a diff in pull request #9832: [multistage] support sort push-down

2022-11-22 Thread GitBox
61yao commented on code in PR #9832: URL: https://github.com/apache/pinot/pull/9832#discussion_r1029934624 ## pinot-query-runtime/src/test/resources/queries/OrderBy.json: ## @@ -0,0 +1,221 @@ +{ + "basic_order_by": { +"tables": { + "basic": { +"schema": [ +

[GitHub] [pinot] Jackie-Jiang commented on issue #9820: Numerical type conversion of literal values in IN and NOT_IN clause

2022-11-22 Thread GitBox
Jackie-Jiang commented on issue #9820: URL: https://github.com/apache/pinot/issues/9820#issuecomment-1324399426 Not all expressions work though. If the result of the expression is type `INT`, then it won't work, and the current optimizer cannot fix it -- This is an automated message from

[GitHub] [pinot] amrishlal commented on issue #9820: Numerical type conversion of literal values in IN and NOT_IN clause

2022-11-22 Thread GitBox
amrishlal commented on issue #9820: URL: https://github.com/apache/pinot/issues/9820#issuecomment-1324397213 > After a second thought, should we consider handling it in the PredicateEvaluatorFactory where the data type is already known? That way we can also handle arbitrary expressions on

[GitHub] [pinot] Jackie-Jiang commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

2022-11-22 Thread GitBox
Jackie-Jiang commented on issue #9812: URL: https://github.com/apache/pinot/issues/9812#issuecomment-1324396500 Actually this is not an aggregation group-by. Is the intention to get the distinct values? It didn't fail in `0.10.0` probably because the `GROUP BY` is ignored? To get the

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9802: Add memory optimized dimension table

2022-11-22 Thread GitBox
Jackie-Jiang commented on code in PR #9802: URL: https://github.com/apache/pinot/pull/9802#discussion_r1029878916 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/offline/MemoryOptimizedDimensionTable.java: ## @@ -0,0 +1,71 @@ +/** + * Licensed to the Apache

[GitHub] [pinot] 61yao commented on a diff in pull request #9832: [multistage] support sort push-down

2022-11-22 Thread GitBox
61yao commented on code in PR #9832: URL: https://github.com/apache/pinot/pull/9832#discussion_r1029892679 ## pinot-query-planner/src/main/java/org/apache/pinot/query/planner/logical/IntExprRexVisitor.java: ## @@ -0,0 +1,126 @@ +/** + * Licensed to the Apache Software

[pinot] branch master updated: Add max merger and min mergers for partial upsert (#9665)

2022-11-22 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new d62a867d46 Add max merger and min mergers for

[GitHub] [pinot] Jackie-Jiang merged pull request #9665: Add max merger and min mergers for partial upsert

2022-11-22 Thread GitBox
Jackie-Jiang merged PR #9665: URL: https://github.com/apache/pinot/pull/9665 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [pinot] codecov-commenter commented on pull request #9846: [multistage] test coverage for CASE and fix bug with CHAR literals

2022-11-22 Thread GitBox
codecov-commenter commented on PR #9846: URL: https://github.com/apache/pinot/pull/9846#issuecomment-1324362961 # [Codecov](https://codecov.io/gh/apache/pinot/pull/9846?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=The+Apache+Software+Foundation) Report

[GitHub] [pinot] agavra opened a new pull request, #9846: [multistage] test coverage for CASE and fix bug with CHAR literals

2022-11-22 Thread GitBox
agavra opened a new pull request, #9846: URL: https://github.com/apache/pinot/pull/9846 - tests various types for CASE WHEN statements - tests nested CASE WHEN statements with aggs/joins - fixes a bug with literals and ' '-padding behavior -- This is an automated message from the

[GitHub] [pinot] mndoping1 commented on issue #9845: Allow Pinotdb client to hit v2 engine

2022-11-22 Thread GitBox
mndoping1 commented on issue #9845: URL: https://github.com/apache/pinot/issues/9845#issuecomment-1324322101 closing as created the issue in approprate repository here: https://github.com/python-pinot-dbapi/pinot-dbapi/issues/52 -- This is an automated message from the Apache Git

[GitHub] [pinot] mndoping1 closed issue #9845: Allow Pinotdb client to hit v2 engine

2022-11-22 Thread GitBox
mndoping1 closed issue #9845: Allow Pinotdb client to hit v2 engine URL: https://github.com/apache/pinot/issues/9845 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [pinot] ankitsultana commented on a diff in pull request #9808: [multistage] Add Multi Stage Strict Replica Group Routing Strategy

2022-11-22 Thread GitBox
ankitsultana commented on code in PR #9808: URL: https://github.com/apache/pinot/pull/9808#discussion_r1029878486 ## pinot-broker/src/test/java/org/apache/pinot/broker/routing/instanceselector/InstanceSelectorTest.java: ## @@ -814,14 +837,121 @@ public void

[GitHub] [pinot] Jackie-Jiang commented on issue #9788: Logging Limit

2022-11-22 Thread GitBox
Jackie-Jiang commented on issue #9788: URL: https://github.com/apache/pinot/issues/9788#issuecomment-1324317790 On the server side, we have `pinot.query.scheduler.query.log.maxRatePerSecond` which is similar to the config on the broker side. More configs can be found here:

[GitHub] [pinot] mndoping1 opened a new issue, #9845: Allow Pinotdb client to hit v2 engine

2022-11-22 Thread GitBox
mndoping1 opened a new issue, #9845: URL: https://github.com/apache/pinot/issues/9845 Python's Pinotdb client version 0.4.5 does not support `table aliasing`. For example, queries such as `SELECT "tbl1"."x" FROM "y" "tbl1"` aren't able to run. However, Pinot's v2 engine allows table

[GitHub] [pinot] jadami10 opened a new pull request, #9844: add table sort function for table size

2022-11-22 Thread GitBox
jadami10 opened a new pull request, #9844: URL: https://github.com/apache/pinot/pull/9844 addresses #9843 this is a `ui` `bugfix` This is probably not the greatest way to do this. Transforming the data from bytes -> humanized -> bytes just to sort it feels hacky. But short of

[GitHub] [pinot] KKcorps commented on issue #9840: Everytime I add a table

2022-11-22 Thread GitBox
KKcorps commented on issue #9840: URL: https://github.com/apache/pinot/issues/9840#issuecomment-1324271076 Agreed. Will add the update functionality in the command. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [pinot] agavra commented on pull request #9832: [multistage] support sort push-down

2022-11-22 Thread GitBox
agavra commented on PR #9832: URL: https://github.com/apache/pinot/pull/9832#issuecomment-1324225162 > Is it possible we re-arrange columns in intermediate stage instead if we pass data schema to intermediate stage? This way we can leave the leaf stage untouched. @61yao no it isn't,

[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9804: Add a new server directory to store metadata

2022-11-22 Thread GitBox
Jackie-Jiang commented on code in PR #9804: URL: https://github.com/apache/pinot/pull/9804#discussion_r1029762721 ## pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java: ## @@ -91,6 +91,7 @@ public abstract class BaseTableDataManager implements

[pinot] branch master updated: Fix Spark connector empty datatable handling in GRPC reader (#9837)

2022-11-22 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 32314bb433 Fix Spark connector empty datatable

[GitHub] [pinot] Jackie-Jiang merged pull request #9837: [pinot-spark-connector] Fix empty data table handling in GRPC reader

2022-11-22 Thread GitBox
Jackie-Jiang merged PR #9837: URL: https://github.com/apache/pinot/pull/9837 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [pinot] Jackie-Jiang commented on issue #9840: Everytime I add a table

2022-11-22 Thread GitBox
Jackie-Jiang commented on issue #9840: URL: https://github.com/apache/pinot/issues/9840#issuecomment-1324156875 Link #9834 which is for the same topic -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [pinot] Jackie-Jiang commented on issue #9840: Everytime I add a table

2022-11-22 Thread GitBox
Jackie-Jiang commented on issue #9840: URL: https://github.com/apache/pinot/issues/9840#issuecomment-1324154183 We should either add a `-update` flag in the `AddTableCommand` or provide a `UpdateTableCommand` to update the table. Also, seems we also need the `DeleteTableCommand`. cc

[GitHub] [pinot] 61yao commented on a diff in pull request #9832: [multistage] support sort push-down

2022-11-22 Thread GitBox
61yao commented on code in PR #9832: URL: https://github.com/apache/pinot/pull/9832#discussion_r1029748399 ## pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/plan/ServerRequestPlanVisitor.java: ## @@ -90,6 +92,9 @@ public static ServerPlanRequestContext

[GitHub] [pinot] Jackie-Jiang commented on issue #9839: Improve AND filter operator

2022-11-22 Thread GitBox
Jackie-Jiang commented on issue #9839: URL: https://github.com/apache/pinot/issues/9839#issuecomment-1324150138 @ashishkf Good point! There are actually some discussion about providing reversed doc iterator in #8979. We should consider adding a method to return the reversed iterator, and

[GitHub] [pinot] 61yao commented on pull request #9832: [multistage] support sort push-down

2022-11-22 Thread GitBox
61yao commented on PR #9832: URL: https://github.com/apache/pinot/pull/9832#issuecomment-1324147001 > @Jackie-Jiang - yes, the V2 engine requires that the server returns columns in the order that would be the result of the logical plan (and also that it doesn't have extra or missing

[GitHub] [pinot] Jackie-Jiang commented on issue #9393: delete segment(s) api for an offline table doesn't delete segments from server disc

2022-11-22 Thread GitBox
Jackie-Jiang commented on issue #9393: URL: https://github.com/apache/pinot/issues/9393#issuecomment-1324144928 This is a known issue, which is tracked here #9102 Is the problem resolved after removing these 2 extra entries? -- This is an automated message from the Apache Git

[GitHub] [pinot] agavra commented on pull request #9832: [multistage] support sort push-down

2022-11-22 Thread GitBox
agavra commented on PR #9832: URL: https://github.com/apache/pinot/pull/9832#issuecomment-1324124148 @Jackie-Jiang - yes, the V2 engine requires that the server returns columns in the order that would be the result of the logical plan (and also that it doesn't have extra or missing

[GitHub] [pinot] Jackie-Jiang commented on issue #9843: In Pinot dashboard, sorting on table size takes alphabetical order

2022-11-22 Thread GitBox
Jackie-Jiang commented on issue #9843: URL: https://github.com/apache/pinot/issues/9843#issuecomment-1324066979 cc @joshigaurava @jayeshchoudhary @jadami10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [pinot] Jackie-Jiang opened a new issue, #9843: In Pinot dashboard, sorting on table size takes alphabetical order

2022-11-22 Thread GitBox
Jackie-Jiang opened a new issue, #9843: URL: https://github.com/apache/pinot/issues/9843 It should be size order instead with regard to unit (e.g. GB before MB) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [pinot] siddharthteotia commented on pull request #9727: Thread Level Usage Accounting and Query Killing on Server

2022-11-22 Thread GitBox
siddharthteotia commented on PR #9727: URL: https://github.com/apache/pinot/pull/9727#issuecomment-1324035346 @jasperjiaguo might want to share some example validation we did internally on PROD use case to demonstrate killing. Let's also make sure to address any comments that come

[GitHub] [pinot] siddharthteotia merged pull request #9727: Thread Level Usage Accounting and Query Killing on Server

2022-11-22 Thread GitBox
siddharthteotia merged PR #9727: URL: https://github.com/apache/pinot/pull/9727 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[pinot] branch master updated: Download offline segments from peers (#9710)

2022-11-22 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pinot.git The following commit(s) were added to refs/heads/master by this push: new 6cfa6dca8b Download offline segments from

[GitHub] [pinot] chenboat merged pull request #9710: Download offline segments from peers

2022-11-22 Thread GitBox
chenboat merged PR #9710: URL: https://github.com/apache/pinot/pull/9710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [pinot] gortiz commented on pull request #9842: [draft] Add chronicle bytes alternative to LArray.

2022-11-22 Thread GitBox
gortiz commented on PR #9842: URL: https://github.com/apache/pinot/pull/9842#issuecomment-1323953675 This first commit is a draft. Tests need to be extended to cover the new implementation and I would also add at least one benchmark to verify that there is no performance degradation. It

[GitHub] [pinot] gortiz opened a new pull request, #9842: [draft] Add chronicle bytes alternative to LArray.

2022-11-22 Thread GitBox
gortiz opened a new pull request, #9842: URL: https://github.com/apache/pinot/pull/9842 This PR explores the possibility of using [Chronicle Bytes](https://github.com/OpenHFT/Chronicle-Bytes) as an implementation of PinotDataBuffer instead of LArray. The main reason to do that is to be

[GitHub] [pinot] vmarchaud commented on issue #9812: bug(broker): fail to group by after groovy extraction when upgrading to 0.11

2022-11-22 Thread GitBox
vmarchaud commented on issue #9812: URL: https://github.com/apache/pinot/issues/9812#issuecomment-1323886488 @Jackie-Jiang I'm using the default (https://github.com/apache/pinot/blob/master/pinot-tools/src/main/resources/conf/log4j2.xml) log4j and i can't find any stacktrace in the log

[GitHub] [pinot] 61yao commented on pull request #9836: [multistage][POC] Move mailbox instance from global map to request context

2022-11-22 Thread GitBox
61yao commented on PR #9836: URL: https://github.com/apache/pinot/pull/9836#issuecomment-1323298327 I took a look again about the fairness scheduling. This PR has nothing to do with that because it only deals with sending mailbox instead of receiving. I agree receiving side needs more