#general
@ravishankar.nair: @ravishankar.nair has joined the channel
@ravishankar.nair: Joined Pinot Slack Channel....
#random
@ravishankar.nair: @ravishankar.nair has joined the channel
#troubleshooting
@brianolsen87: same issue...i'm gonna update the default null values I think...it's only for a demo
@mayanks: If also try the default null value of 0 millisSinceEpoch.
@brianolsen87: ```Driver, record read time : 50771 Driver, stats collector time : 0 Driver, indexing time : 20623 Tarring segment from: /tmp/pinot-9a3c12a1-8374-45b0-9d12-7fbd8a2a2dee/output/covid_cases_OFFLINE_1971/01/01_2021/03/30_0 to: /tmp/pinot-9a3c12a1-8374-45b0-9d12-7fbd8a2a2dee/output/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0.tar.gz Size for segment: covid_cases_OFFLINE_1971/01/01_2021/03/30_0, uncompressed: 135.04M, compressed: 52.17M Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS Start pushing segments: [/tmp/pinot-quick-start/segments/covid_cases_OFFLINE_1971/01/01_2021/03/30_0.tar.gz]... to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@2d778add] for table covid_cases Pushing segment: 30_0 to location:
@mayanks: The log above indicates that the push was successful. Do you have access to server logs to see what happeneed when the server tried to load it?
@brianolsen87: Kind of a confusing message: > org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Failed to get reason" while sending request: <http://<ip-address>:9000/segments/covid_cases/covid_cases_OFFLINE_1971%2F01%2F01_2021%2F03%2F30_0>
@jackie.jxt: The segment is rejected because `1970/01/01` is not in the valid time range. We allow time from `1971/01/01` to `2071/01/01` to prevent wrong format time being ingested
@fx19880617: I think this segment is successfully uploaded ,so you can find it through ui
@fx19880617: server log shows that it’s failed to download the segment from the controller: ```Got temporary error status code: 500 while downloading segment from:
@fx19880617: wondering where you run this setup? seems local deployment?
@fx19880617: can you also try to make a call to ```
@fx19880617: Push job says ```Pushing segment: 30_0 to location:
@fx19880617: then can you check segment metadata and see the downloadURL there
@fx19880617: also for pinot controller ,can you check if u have this config in the `values.yaml` file: ```pinot.set.instance.id.to.hostname=true``` It should make pinot controller use hostname other than ip address to register itself
@brianolsen87: @jackie.jxt yeah, I eventually fixed that to 1971 and then got this issue. As @fx19880617 points out it is updated in the UI to see the segment but it never downloads.
@brianolsen87: This is running on an ec2 host running docker compose and sharing resources with mongo/Trino so might as well be a local setup :slightly_smiling_face:
@brianolsen87: I’ll try what you said in a moment Xiang! Thanks :pray:
@brianolsen87: @fx19880617, I just noticed that you said this. > so I think this means you are still on k8s. I'm trying to set this up in docker compose right now...what is the default for this property if I haven't set it for other nodes? `controller.helix.cluster.name`
@brianolsen87: Here are the settings in docker compose ``` pinot-controller: image: apachepinot/pinot:latest hostname: pinot-controller volumes: - ./pinot-docker-demo/pinot/controller:/tmp/data/controller - ./conf:/opt/pinot/conf ports: - "9000:9000" command: StartController -configFileName /opt/pinot/conf/pinot-controller.conf depends_on: - zookeeper - kafka networks: - trino-network pinot-broker: image: apachepinot/pinot:latest hostname: pinot-broker ports: - "8099:8099" command: StartBroker -zkAddress zookeeper:2181 depends_on: - zookeeper - kafka - pinot-controller networks: - trino-network pinot-server: image: apachepinot/pinot:latest hostname: pinot-server volumes: - ./pinot-docker-demo/pinot/server:/tmp/data/server ports: - "8098:8098" command: StartServer -zkAddress zookeeper:2181 depends_on: - zookeeper - kafka - pinot-controller networks: - trino-network```
@fx19880617: oic, then in this `./conf/pinot-controller.conf`, can you add that config : `
@fx19880617: also for `pinot-broker.conf` and `pinot-server.conf`
@brianolsen87: got it, will add for all :slightly_smiling_face:
@fx19880617: Sure. You can verify by looking at the instance name. It should give Pinot-controller instead of IP address
@brianolsen87: id is now ```{ "id": "Controller_pinot-controller_9000",``` running the batch job again :crossed_fingers:
@fx19880617: :crossed_fingers:
@brianolsen87: hmm, still no luck...
@brianolsen87: segment is still showing up as bad...I think I must be doing something off like one setting I missed or something
@fx19880617: hmm what’s the download uri this time?
@brianolsen87: ```Copying segment from /var/pinot/controller/data/pinot-controller_9000/fileUploadTemp/tmp-f4f3ebd3-0a20-4438-a303-940dd63e4aa5 to file:/var/pinot/controller/data/covid_cases/covid_cases_OFFLINE_2020%2F10%2F23_2021%2F01%2F28_0 Moved segment covid_cases_OFFLINE_2020/10/23_2021/01/28_0 from temp location /var/pinot/controller/data/pinot-controller_9000/fileUploadTemp/tmp-f4f3ebd3-0a20-4438-a303-940dd63e4aa5 to /var/pinot/controller/data/covid_cases/covid_cases_OFFLINE_2020/10/23_2021/01/28_0 Added segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 of table: covid_cases_OFFLINE to property store Assigning segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 with instance partitions: {"instancePartitionsName":"covid_cases_OFFLINE","partitionToInstancesMap":{"0_0":["Server_pinot-server_8098"]}} for table: covid_cases_OFFLINE Assigned segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 to instances: [Server_pinot-server_8098] for table: covid_cases_OFFLINE Assigning segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 to instances: [Server_pinot-server_8098] for table: covid_cases_OFFLINE Added segment: covid_cases_OFFLINE_2020/10/23_2021/01/28_0 to IdealState for table: covid_cases_OFFLINE Handled request from 172.18.0.9 POST
@brianolsen87: hmmm
@brianolsen87: Anything off in my table config? ```{ "OFFLINE": { "tableName": "covid_cases_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { "timeType": "MILLISECONDS", "schemaName": "covid_cases", "timeColumnName": "cdc_report_dt", "replication": "1" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false, "aggregateMetrics": false, "nullHandlingEnabled": true, "invertedIndexColumns": [], "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "loadMode": "MMAP" }, "metadata": {}, "isDimTable": false } }```
@fx19880617: ok, so it means it’s still server cannot download it ?
@fx19880617: looks ok
@fx19880617: can you click segment metadata and look at the downloadURI field
@fx19880617: I think that one should be fixed as well
@brianolsen87: I try clicking on segment and it fails
@brianolsen87: brings me to the homescreen
@brianolsen87: I have to run out for 15 min i'll be right back
@fx19880617: hmm it should send you to this:
@fx19880617: and shows ```"segment.offline.download.url": "
@fx19880617: also can you try to set `timeType` to `SIMPLE_DATE_FORMAT:yyyy/MM/dd` in your table config
@brianolsen87: had to take this offline with xiang..will update with answer
@brianolsen87: I am curious to know what the solution would be here though if anyone knows
@jackie.jxt: Do all these time fields contain `null` value? Pinot does not allow `null` value in the main time field (the one you put in the `table config`)
@jackie.jxt: If you just want to ingest the data, you can simply store them as dimensions without defining the time column
@brianolsen87: Now they default to a value so that shouldn't be the issue but that’s good to know. If I remove these from the time fields would we not be taking advantage of indexing based on time date time fields?
@jackie.jxt: There is no special indexing on time field currently, so if you don’t need retention management, storing them as dimensions or date times should have the same performance
@brianolsen87: ah good to know...thanks Jackie
@pabraham.usa: Hello Anyone tried running Pino in arm-based chip machines? currently I am using x86 and looking to migrate to arm. Not sure there is an arm based Pino docker image?
@fx19880617: Please give it a try to build with a different base image. The Dockerfile is inside the code base
@pabraham.usa: Thanks @fx19880617, will give a try
@fx19880617: Please let me know if you see any issue there. Also please contribute if you get it working ;)
@ravishankar.nair: @ravishankar.nair has joined the channel
@sosyalmedya.oguzhan: ```server: name: server ports: netty: 8098 admin: 8097 replicaCount: 1 dataDir: /var/pinot/server/data/index segmentTarDir: /var/pinot/server/data/segment persistence: enabled: true accessMode: ReadWriteOnce size: "30Gi" mountPath: /var/pinot/server/data storageClass: "alicloud-disk-available" #storageClass: "ssd" jvmOpts: "-Xms512M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/opt/pinot/gc-pinot-server.log -Dplugins.include=pinot-hdfs" log4j2ConfFile: /opt/pinot/conf/pinot-server-log4j2.xml pluginsDir: /opt/pinot/plugins service: annotations: {} clusterIP: "" externalIPs: [] loadBalancerIP: "" loadBalancerSourceRanges: [] type: ClusterIP port: 8098 nodePort: "" resources: {} nodeSelector: {} affinity: {} tolerations: [] podAnnotations: {} updateStrategy: type: RollingUpdate # Extra configs will be appended to pinot-server.conf file extra: configs: |- pinot.set.instance.id.to.hostname=true pinot.server.instance.realtime.alloc.offheap=true pinot.server.instance.enable.split.commit=true pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS pinot.server.storage.factory.hdfs.hadoop.conf.path=/opt/hadoop/conf/ pinot.server.segment.fetcher.protocols=file,http,hdfs pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher``` here is my server config. I want to use hdfs as deep storage. I put related hdfs jars under the pinot, but server pod not starting when i add the `-Dplugins.include=pinot-hdfs`
@fx19880617: by default it should load all the plugins
@fx19880617: also ```pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` should have same indentation as previous configs
@fx19880617: if you specify ```-Dplugins.include=pinot-hdfs``` then you may also need to specify all other plugins e.g, ```-Dplugins.include=pinot-hdfs,pinot-avro,pinot-parquet```
@fx19880617: what’s the exceptions in the logs?
@sosyalmedya.oguzhan: so, we do not need to pass -Dplugins.include ? indentation seems correct if i am not wrong? i can not see any exception, because server pods is crashing, it's restarting continuously
@fx19880617: indentation is correct, it’s my broswer issue, :stuck_out_tongue:
@fx19880617: try not to pass `-Dplugins.include`
@fx19880617: and see what you get from the server log
@fx19880617: might be some lib conflict I feel
@sosyalmedya.oguzhan: ok, i'm trying
@fx19880617: then we may need to re-shade the pinot-hdfs jar
@fx19880617: internally pinot-hdfs refers to hadoop-common 2.7.0
@sosyalmedya.oguzhan: actually, i can run ingestion job with hdfs deep storage. i'm not sure about lib conflict
@sosyalmedya.oguzhan: it's worked now, i don't know how i did fix it :smile:
@fx19880617: :stuck_out_tongue:
@fx19880617: so I guess restart make the trick
@fx19880617: please keep the server log and share next time (hope there is no next time as well)
@dlavoie: All plugins are loaded by default, the plugin dir doesn’t need to be customized
#complex-type-support
@g.kishore: @steotia ^^
@steotia: @steotia has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
