#general
@ysuo: pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars
@mayanks: Using `/tmp` is not a good idea
@ysuo: for server node, the above properties are used to config local attached disk, right?
@mayanks: Yes, set these to a local attached disk mount.
@ysuo: according to the doc
@ysuo: Got it, thanks, @mayanks
@mayanks: Yes, for OSS Pinot, you need both.
@mayanks: Deep store is for storing copy of data for disaster recovery. Local copy is needed for faster processing.
@ysuo: how much storage should i set for my local copy, is there a resource that i can refer to?
@mayanks: Depends on your workload requirements. You can go up to 1 TB per node
@ysuo: I see, thanks, Mayank.
@ysuo: and if i want to use EBS, i should set two properties referred to EBS?
@ysuo: and the following properties are used to set deepstore:
@ysuo: pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS 8 pinot.server.storage.factory.s3.region=us-west-2 9 pinot.server.segment.fetcher.protocols=file,http,s3 10 pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
@ysuo: pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.server.storage.factory.s3.region=us-west-2 pinot.server.segment.fetcher.protocols=file,http,s3 pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
@mayanks: Deepstore and local attached disk have different purposes for pinot server (you need both, and not either/or). And yes you will need to set the properties you mentioned.
@mayanks: You can refer to:
@ysuo: thanks
@ysuo: If i set retentionTimeUnit to 30 DAYS in a table config, will controller periodic task delete data in S3 deep store which is older than 30 days too?
@ysuo: By default, no retention is set. In this case, the data in the server local storage will grow increasingly? And if so, what happens if there is not enough space for local storage?
@mayanks: Yes old data is deleted from deep store as well. You don’t want to run out of storage space. There are metrics to monitor and alert on that
@kaushalaggarwal349: @kaushalaggarwal349 has joined the channel
@robert.gustafsson1: @robert.gustafsson1 has joined the channel
@surajkmth29: Hi Folks, I'm trying to load test Apache pinot with jmeter, From where can I download the PINOT JDBC driver jar ? I have included the pinot-jdbc-client-0.9.3.jar in the JMETER path however it's resulting in internal dependency related errors ... Is there a PINOT JDBC Driver jar that i could use directly ?
@mayanks: Please use Pinot Java client not JDBC driver
@ysuo: Hi, I’m a little confused about the instance name. If i specify a instanceId for a server, take pinot-server for example, the instance name will be Server_pinot-server_port. So what’s the best practice for setting up a pinot cluster with 3 or more servers using Docker or Kubernetes? Should i set a unique instanceId for each one of them or maybe there is a better way?
@mayanks: Yes, you’ll need to have unique ids for them
@ysuo: Useful to know it.:grin:
@ysuo: In my test, some properties can be set like this, *pinot-admin.sh StartController -zkAddress localhost:2191 -controllerPort 9000.*
@ysuo: My question is, could every single one of the properties in a config file be set in the above way?
@mayanks: Yes for the ones you need
@sboggavarapu: @sboggavarapu has joined the channel
@sboggavarapu: :wave: Hi everyone! Nice to be here.
@sboggavarapu: So, I am trying to get a table configured for upsert. So, I cannot use star tree index as it says in docs? Does that mean I am only limited to use any other index other than star tree?
@sboggavarapu: Also, according to docs, it is not guaranteed that order is preserved in case there is any sort index on any column.
@sboggavarapu: ``` When two records of the same primary key are ingested, the record with the greater event time (as defined by the time column) is used. When records with the same primary key and event time, then the order is not determined. In most cases, the later ingested record will be used, but may not be so in the cases when the table has a column to sort by.```
@sboggavarapu: Does that mean , I cannot use that index too if I want upsert to be configured for a table?
@mayanks: Yes, StarTree Index is not supported with upsert. But note that startree index helps in case of really low selectivity queries (millions of rows selected by the query. And you can still get great perf from other indexes.
@mayanks: You can use sorted index with upsert.
@sboggavarapu: Thanks
@kavin.kuppusamy2: @kavin.kuppusamy2 has joined the channel
#random
@kaushalaggarwal349: @kaushalaggarwal349 has joined the channel
@robert.gustafsson1: @robert.gustafsson1 has joined the channel
@sboggavarapu: @sboggavarapu has joined the channel
@kavin.kuppusamy2: @kavin.kuppusamy2 has joined the channel
#feat-presto-connector
@kchavda: @kchavda has joined the channel
#order-by
@jt: @jt has left the channel
#fraud
@kchavda: @kchavda has joined the channel
@kchavda: @kchavda has left the channel
#troubleshooting
@kaushalaggarwal349: @kaushalaggarwal349 has joined the channel
@robert.gustafsson1: @robert.gustafsson1 has joined the channel
@diana.arnos: Hello again :grimacing: How can I setup S3 as deep storage while using the helm chart? I tried adding the configs from
@diana.arnos: ```# Source: pinot/templates/controller/configmap.yaml # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # #
@mark.needham: This config should work
@mark.needham: I think it was @diogo.baeder who shared that
@diogo.baeder: Now I realize that that format is something we use internally, there's an app we have which converts that to Helm templates and then publishes as k8s manifests
@diogo.baeder: But the configs there are relevant, yes
@diana.arnos: We are using ingress and our config works until we add the `extra configs` part. Then the controller pod has nothing listening on port 9000, which is the port the ingress rule uses ```root@responses-pinot-controller-0:/opt/pinot# ss -tnlp State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 3 0.0.0.0:5556 0.0.0.0:* users:(("java",pid=1,fd=30))``` We don't setup AWS access key for we use KIAM containers to manage permissions. This is the config that works (those are staging/dev values, so don't mind the resources): ```image: repository: dianaarnos/pinot tag: 0.9.3-with-ingress metadata: labels: app: responses-pinot
@diana.arnos: These are the configs inside `pinot-controller.conf` ```controller.helix.cluster.name=responses controller.port=9000 controller.data.dir=s3://<redacted> controller.zk.str=responses-pinot-zookeeper:2181 pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.controller.storage.factory.s3.region=eu-west-1 pinot.controller.segment.fetcher.protocols=file,http,s3 pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.set.instance.id.to.hostname=true controller.task.scheduler.enabled=true```
@diana.arnos: I was able to find a working config, after I noticed that @diogo.baeder had a couple of extra configuration lines for both the server and the controller. Those configs do not appear on the doc
@mark.needham: do you know which of those make things work?
@diogo.baeder: Ah, nice, good to know it worked in the end :slightly_smiling_face:
@diana.arnos: @mark.needham, I didn't try each separately yet, for I needed to approve a working config for our staging environment first :grimacing: After I'm able to query our table I can try that
@ahsen.m: can someone please explain why deep storage is needed?
@ahsen.m: i have mongodb > kafka .. now to use pinot do i need deep storage as well?
@mark.needham: the deep store is the permanent storage of pinot segments
@diana.arnos: There's a nice explanation here:
@mark.needham: so you do need to configure it
@ahsen.m: ahh thank u
@ahsen.m: do we support gcp storage or its just s3?
@mark.needham: gcp should work yeh
@mark.needham: or hdbs
@mark.needham: hdfs* even
@ahsen.m: okie awesome, thank u let me look into this
@diana.arnos: Now I have another problem :rolling_on_the_floor_laughing: Whenever I try to create a schema: ```> POST /schemas { "_code": 500, "_error": null }``` ```Starting TaskMetricsEmitter with running frequency of 300 seconds. [TaskRequestId: auto] Start running task: TaskMetricsEmitter [TaskRequestId: auto] Finish running task: TaskMetricsEmitter in 4ms Server error: java.lang.NullPointerException: null at java.util.Objects.requireNonNull(Objects.java:221) ~[?:?] at java.util.Optional.<init>(Optional.java:107) ~[?:?] at java.util.Optional.of(Optional.java:120) ~[?:?] at org.apache.pinot.controller.api.access.AccessControlUtils.validatePermission(AccessControlUtils.java:48) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.apache.pinot.controller.api.resources.PinotSchemaRestletResource.addSchema(PinotSchemaRestletResource.java:215) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:353) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at java.lang.Thread.run(Thread.java:829) [?:?]```
@diana.arnos: Ok, it stopped... all by itself. Magic.
@ahsen.m: Diana, could you please share the yaml file for s3 that fixed the issue?
@sboggavarapu: @sboggavarapu has joined the channel
@diana.arnos: Now I have another problem :rolling_on_the_floor_laughing: Whenever I try to create a schema: ```> POST /schemas { "_code": 500, "_error": null }``` ```Starting TaskMetricsEmitter with running frequency of 300 seconds. [TaskRequestId: auto] Start running task: TaskMetricsEmitter [TaskRequestId: auto] Finish running task: TaskMetricsEmitter in 4ms Server error: java.lang.NullPointerException: null at java.util.Objects.requireNonNull(Objects.java:221) ~[?:?] at java.util.Optional.<init>(Optional.java:107) ~[?:?] at java.util.Optional.of(Optional.java:120) ~[?:?] at org.apache.pinot.controller.api.access.AccessControlUtils.validatePermission(AccessControlUtils.java:48) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.apache.pinot.controller.api.resources.PinotSchemaRestletResource.addSchema(PinotSchemaRestletResource.java:215) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:353) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-de4cb2cc01d611c92ff5d1fc6a49ee9f8b113192] at java.lang.Thread.run(Thread.java:829) [?:?]```
@ryantle1028: hi guys i have multiple kafka sasl separate Kerberos. i don't know what parameter on real time table use for point to krb5.conf or content inside krb5.conf. i need to config realtime tables and multiple kdc. Please recommend which parameter or some example use for that.
@mayanks: @slack1 any suggestions ^^
@slack1: this one is for @chinmay.cerebro
@chinmay.cerebro: I haven’t played around with Kerberos config in Pinot myself . Can’t give any quick answer I’m afraid but can you share the error you’re seeing ? Also I’m assuming you’ve seen this for Kafka client config :
@chinmay.cerebro: If you have a working Kafka client the same config should work with Pinot
@chinmay.cerebro: My suggestion is to get the client working first
@luisfernandez: hey friends... I was issuing rolling updates for pinot-servers with `kubectl` however I noticed that when I run this command I always get a brand new server and have to issue rebalances again, is restarting servers something that requires rebalancing? I'm pretty sure it must be something funky going on with our config
@dlavoie: How have you deployed the cluster in the first place?
@luisfernandez: I’m using GKE, with the helm chart provided in the OSS
@dlavoie: and what command do you use to update your servers?
@luisfernandez: what do you mean to update my severs? to restart them you mean?
@luisfernandez: i just do `kubectl rollout restart statefulset/pinot-server`
@dlavoie: > I was issuing rolling updates for pinot-servers with kubectl Could you provide the specific of what you did here.
@dlavoie: and could you describe what you mean by: > I always get a brand new server
@luisfernandez: when i do that then i get 2 new servers and things start to look like this:
@dlavoie: Ok
@dlavoie: Check if you have `
@luisfernandez: lemme check
@dlavoie: you would need that property for all components
@dlavoie: With this, your pods will keep the same identity in Helix.
@luisfernandez: all components need that yes?
@luisfernandez: `pinot.set.instance.id.to.hostname`
@dlavoie: servers for sure
@luisfernandez: oh yes i def don’t see that being set for the server
@dlavoie: let me check other nodes
@dlavoie: Yes, they all need it. controller, minion, server and broker
@luisfernandez: yep they all have it in my case except for the server
@luisfernandez: so what does this do in particular?
@dlavoie: Tell Helix that the cluster member should use its OS hostname as an identifier rather than IP/PORT.
@dlavoie: In a k8s statefulset, IP is dynamic, hostname is preditable per pod instance
@luisfernandez: oh i have different environments and in sandbox i do see the name being different
@luisfernandez: like this
@dlavoie: Making it a default value in the helm chart would be a sensible thing to do. If you want to contribute to the OSS helm chart, that is an easy PR.
@luisfernandez:
@luisfernandez: i think that was by default what the helm chart did i feel like we change the configs but i will double check
@luisfernandez: now it makes sense cause i remember when i used to do rolling updates in sandbox i wouldn’t have to do any of the things i was doing now in the new env lol
@luisfernandez: rolling restarts* sorry
@luisfernandez: thank you so much for your help
@dlavoie: You’re welcome :+1:
@lakshmanan.velusamy: Hi Community, Can the dimension tables be created across different tenants with the same name ?
@lakshmanan.velusamy: cc @weixiang.sun @mingfeng.tan
@mayanks: Table names need to be unique within a Pinot cluster.
@mingfeng.tan: To my understanding, a table can only be associated with one tenant. If tenant is not specified, then it goes to DeafultTeant. But since dimension table normally is small, it’s would be fine to duplicate with it in another tenant but with different name.
@mingfeng.tan: Correct me if I’m wrong.
@mayanks: Yes, it can be duplicated (with different name of course). You can still have a single push job that can push data to multiple tables.
@lakshmanan.velusamy: With hybrid table, the lookup() query becomes tricky if the realtime and offline tables are in different tenants (as the dimension table names are different).
@mayanks: But you can have multiple tenants for a server/broker. So you could create a dedicated tenant for dim table and mark the realtime and offline servers with that additional tenant.
@kavin.kuppusamy2: @kavin.kuppusamy2 has joined the channel
@ahsen.m: so i updated values like ``` persistence: enabled: true accessMode: ReadWriteOnce size: 2G mountPath: /var/pinot/controller/data storageClass: "" extraVolumes: - name: gcp-credentials-volume secret: secretName: gcp-credentials items: - key: gcp_creds_json path: gcp_credentials.json extraVolumeMounts: - name: gcp-credentials-volume mountPath: /opt/pinot/gcp readOnly: true but when i run helm template testing --debug . the template it generates does not have any volume mount named `gcp-credentials-volume`, any idea's?```
@sunhee.bigdata: Hi, I was issuing partitioning in Pinot. When I query 'select where in' partition column, It doesn't show any record. But when I query 'select where not in' partirion column, It seems ok. And after flushing segment, query 'select where in' result in right record. but after producing row (before flushed segments), it doesn't show record *)realtime table *)partition column : subject *) kafka topic partiruons = 4 *) pinot partitiom function confif : Murmur
#pinot-dev
@stuart.millholland: What tools have you used to load test pinot? We are thinking jmeter, but curious if what others might have used.
@stuart.millholland: We want to run queries as tenants (hundreds of tenants) and our use-case is in the max hundreds of queries per second.
@atri.sharma: Jmeter is good
@jadami: i use python
@richard892: I prefer gatling
@richard892: Jmeter has had various issues with the way it records latencies historically
@mayanks: There is also QueryRunner in Pinot code base
#announcements
@kchavda: @kchavda has joined the channel
#roadmap
@kchavda: @kchavda has joined the channel
#presto-pinot-connector
@kchavda: @kchavda has joined the channel
#getting-started
@kaushalaggarwal349: @kaushalaggarwal349 has joined the channel
@robert.gustafsson1: @robert.gustafsson1 has joined the channel
@sboggavarapu: @sboggavarapu has joined the channel
@kavin.kuppusamy2: @kavin.kuppusamy2 has joined the channel
#releases
@kchavda: @kchavda has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
