Apache Pinot Daily Email Digest (2021-04-19)

Pinot Slack Email Digest Mon, 19 Apr 2021 19:00:37 -0700

#general

@chxing: Hi all, We want to build a new env for pinot, we need to build some hybrid tables, Can we assign realtime table data to specified vms and assign offline table data to other vms? thx
@mayanks: Yes you can use tags to assign different tags to RT and offline nodes. Also you can use hdfs as deep store
@chxing: Thx *Mayank, if we use hadoop as deep store, how about the performance compared to use disk directly*
@mayanks: Deep store is not in read or write critical path. It is only used as a backup store
@chxing: So your means, we can only use *disk directly as pinot storage right?*
@chxing: for query and ingest
@mayanks: Yes
@chxing: Thx Mayank
@chxing: Another question, can pinot use hadoop hdfs as storage?
@g.kishore: Yes, for backup
@hoyajigi: @hoyajigi has joined the channel
@filip: @filip has joined the channel
@dungdm93: @dungdm93 has joined the channel
@nikhil.sulegaon: @nikhil.sulegaon has joined the channel
@ioannis: @ioannis has joined the channel
@tyler: @tyler has joined the channel
@mmiottobarbosa: @mmiottobarbosa has joined the channel

#random

@hoyajigi: @hoyajigi has joined the channel
@filip: @filip has joined the channel
@dungdm93: @dungdm93 has joined the channel
@nikhil.sulegaon: @nikhil.sulegaon has joined the channel
@ioannis: @ioannis has joined the channel
@tyler: @tyler has joined the channel
@mmiottobarbosa: @mmiottobarbosa has joined the channel

#troubleshooting

@ravikumar.m: Dear All, I am trying a PoC which I am trying to ingest data from S3 bucket to pinot local. I got sample table config for that(airlineStats_offline_table_config.json) and I created schema file also. I found from Pinot documentation that for ingestion, I should create a job.yml file and should mention in ingestion command. But I am not able to find any sample job script in incubator-pinot. Can you help me how can create job.yml. any refrence docs or blogs please share. thanks
@jackie.jxt: You should be able to find some job examples in the pinot-tools resource package for the quick-start
@ravikumar.m: Yes, got Thanks @jackie.jxt
@hoyajigi: @hoyajigi has joined the channel
@ravikumar.m: Hi All, In my *S3 bucket* has last one year files exist with the folder structure '*year/month/day*' but in Pinot job spec we can specify the a folder can not cover sub folders(as per Pinot documentation) so how can I specify in my job spec yaml to cover S3 sub folders also. Need your help..
@filip: @filip has joined the channel
@ravikumar.m: Hi, I am getting error when I run `bin/pinot-ingestion-job.sh -jobSpec examples/batch/airlineStats/ingestionJobSpec.yaml` The error: ERROR [LaunchDataIngestionJobCommand] [main] Error: "-jobSpec" is not a valid option Need your help...
@fx19880617: try `-jobSpecFile`
@fx19880617: ```# inputDirURI: Root directory of input data, expected to have scheme configured in PinotFS. inputDirURI: 's3://<my-bucket>/path/to/root/input/data' # includeFileNamePattern: include file name pattern, supported glob pattern. # Sample usage: # 'glob:*.avro' will include all avro files just under the inputDirURI, not sub directories; # 'glob:**/*.avro' will include all the avro files under inputDirURI recursively. includeFileNamePattern: 'glob:**/*.avro'```
@dungdm93: @dungdm93 has joined the channel
@nikhil.sulegaon: @nikhil.sulegaon has joined the channel
@ioannis: @ioannis has joined the channel
@tyler: @tyler has joined the channel
@nikhil.sulegaon: Hey folks! New to the Pinot - super stoked to be here! This might be a repeated question, but I see an `OFFLINE` table in our Pinot cluster has a lot of `BAD` segments. Is there a list of common reasons that might make a segment go `BAD`?
@mayanks: If you have access to the server logs, that should tell you why the segments went into `BAD` state. Typically it is because server is unable to load the segment (OOM, no-disk space, etc)
@nikhil.sulegaon: I see! Thank you for the info @mayanks
@mmiottobarbosa: @mmiottobarbosa has joined the channel