Apache Pinot Daily Email Digest (2021-04-27)

[Pinot Slack Email Digest]

#general

@irakov: @irakov has joined the channel
@badri.srinivasan: @badri.srinivasan has joined the channel
@achraibi: Hello all. I’m considering the use of pinot as a backend for a “business intelligence-like” tool for sensor data, that is: user customizable web facing dashboards + some reporting. I’m wondering if Pinot would be a good fit as my requirements are: • a derived metric definition mechanism that would allow developers to define metrics as a function of other metrics either timestamped or static • a fair amount of updates due to errors that might occur in the data transmission chain and due to data being ingested in batches • an “alternative data” mechanism that would set the value of a given metric A based on some criteria (like the existence of B for the same timestamp) • i need to handle approximately 1 million readings a day per production site. These metrics are transmitted each 5 minutes. Useful information can lie within this level of detail but the majority of the time, dashboards will present daily or hourly aggregates • each ingestion must trigger a set of analysis on the data that will perform actions based on business rules (like alerting, creating an issue ticket, etc...) I’m new to big data technologies and despite reading the documentation, I feel I’m missing a building block somewhere between postgresql and pinot. I’m thus seeking for friendly advice that could point me to the right direction. Thanks all.
  @g.kishore: thanks for providing detailed requirement. I can see Pinot has the primitives needed to achieve most of your requirements but for some of them you will to build something on top of Pinot
  @g.kishore: see the concept of derived metrics in Pinot
  @achraibi: Thank you for your feedback. The web framework relies on a relational database for user management, and portfolio specifications (among others). With the concept of derived metrics, will I be able to define metrics based on data in the relational database?
  @g.kishore: We should probably take a concrete example. We can also move this conversation to troubleshooting channel
@karinwolok1: :wine_glass: Help us welcome our newest community members! :pinot: Hello! :wave: :tada: Tell us who you are and what brought you here! :smiley: @narayanan.nachiappan @irakov @badri.srinivasan @pingkarthickselvaraj @erjcan @wenlong.xiong @vinayakb @arunmuthuravi @lars.zwaan @ashwin @mukeshcsahutech @srirajanivetha @pedro.cls93 @jan.stegink @aiyer @ajaythompson @markandrusroberts @hoyajigi @filip @dungdm93 @nikhil.sulegaon @ioannis @tyler @mmiottobarbosa @john.knappster @vishalashank @agateaux @robersegsal @msirek @shgandhi @hasancan.volaka @btripathy @zhong.chen @raj @kuantian.zhang01 @selvakumarcts @kautsshukla @roniepaolo @grebaza @havjyan
@karinwolok1: If there's anyone in the community that's interested in doing some "Getting Started with X in Apache Pinot" type content for others in the community, please let me know! :slightly_smiling_face:

#random

@irakov: @irakov has joined the channel
@badri.srinivasan: @badri.srinivasan has joined the channel

#feat-text-search

@badri.srinivasan: @badri.srinivasan has joined the channel

#feat-presto-connector

@badri.srinivasan: @badri.srinivasan has joined the channel

#troubleshooting

@irakov: @irakov has joined the channel
@badri.srinivasan: @badri.srinivasan has joined the channel
@mohamed.sultan: Hi team. I'm facing some issue when i tried to connect with GCS deep store Here are the logs. Please take a look and help me on this.
  @mohamed.sultan:
  @mohamed.sultan:
  @mohamed.sultan:
  @mohamed.sultan:
  @mohamed.sultan: these are the configurations added. and mounted the gcs key in /tmp/gcs/keys/gcs-keys.json
  @mohamed.sultan: job spec for gcs
  @mohamed.sultan:
  @dlavoie: The configs are referenced here:
  @dlavoie: My bad, you already have the config
  @dlavoie: Can you try with a trailling slash on `controller.data.dir` ?
  @dlavoie: Oh wait, found the issue
  @dlavoie: It fails when building your credential json
  @dlavoie: ``` at com.google.auth.oauth2.GoogleCredentials.fromStream(GoogleCredentials.java:150) ~[pinot-gcs-0.8.0-SNAPSHOT-shaded.jar:0.8.0-SNAPSHOT-6d34505ad9dcc25c2f48f9aa1b0985171fb0351c]```
  @dlavoie: Your JSON credential is a directory apparently. Not sure how you are running Pinot but if you are using volumes, your credential mount is probably bad
  @mohamed.sultan: i used configmap to mount. But its file :neutral_face:
  @mohamed.sultan: ok got it! let me try with the changes
  @dlavoie: Check inside the container, I’m pretty sure you mounted `gcs.key.json` as a folder and that there’s a nested file for your configmap key inside
  @mohamed.sultan: yeah. you are right
  @mohamed.sultan: the path is like /var/gcs/keys/gcs-key.json/
  @mohamed.sultan: my bad
@kmvb.tau: Hello guys, Data Ingestion not working for data with JSON data types. Followed the below steps in StackOverflow. `````` Even examples given for JSON data types in build also not working () Schema file: ```{ "metricFieldSpecs": [], "dimensionFieldSpecs": [ { "dataType": "STRING", "name": "name" }, { "dataType": "LONG", "name": "age" }, { "dataType": "STRING", "name": "subjects_str" }, { "dataType": "STRING", "name": "subjects_name", "singleValueField": false }, { "dataType": "STRING", "name": "subjects_grade", "singleValueField": false } ], "dateTimeFieldSpecs": [], "schemaName": "myTable" }``` Table Config: ```{ "tableName": "myTable", "tableType": "OFFLINE", "segmentsConfig": { "segmentPushType": "APPEND", "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy", "schemaName": "myTable", "replication": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [], "noDictionaryColumns": [ "subjects_str" ], "jsonIndexColumns": [ "subjects_str" ] }, "metadata": { "customConfigs": {} }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY", "batchConfigMaps": [], "segmentNameSpec": {}, "pushSpec": {} }, "transformConfigs": [ { "columnName": "subjects_str", "transformFunction": "jsonFormat(subjects)" }, { "columnName": "subjects_name", "transformFunction": "jsonPathArray(subjects, '$.[*].name')" }, { "columnName": "subjects_grade", "transformFunction": "jsonPathArray(subjects, '$.[*].grade')" } ] } } Data.json {"name":"Pete","age":24,"subjects":[{"name":"maths","grade":"A"},{"name":"maths","grade":"B--"}]} {"name":"Pete1","age":23,"subjects":[{"name":"maths","grade":"A+"},{"name":"maths","grade":"B--"}]} {"name":"Pete2","age":25,"subjects":[{"name":"maths","grade":"A++"},{"name":"maths","grade":"B--"}]} {"name":"Pete3","age":26,"subjects":[{"name":"maths","grade":"A+++"},{"name":"maths","grade":"B--"}]}``` please help me to rectify this issue. Ingestion Job output: (No error) ```bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile /home/sas/apache-pinot-incubating-0.7.1-bin/examples/batch/jsontype/ingestionJobSpec.yaml SegmentGenerationJobSpec: !!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec cleanUpOutputDir: false excludeFileNamePattern: null executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner, segmentMetadataPushJobRunnerClassName: null, segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner, segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner} includeFileNamePattern: glob:**/*.json inputDirURI: examples/batch/jsontype/rawdata jobType: SegmentCreationAndTarPush outputDirURI: examples/batch/jsontype/segments overwriteOutput: true pinotClusterSpecs: - {controllerURI: ''} pinotFSSpecs: - {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file} pushJobSpec: {pushAttempts: 2, pushParallelism: 1, pushRetryIntervalMillis: 1000, segmentUriPrefix: null, segmentUriSuffix: null} recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.json.JSONRecordReader, configClassName: null, configs: null, dataFormat: json} segmentCreationJobParallelism: 0 segmentNameGeneratorSpec: null tableSpec: {schemaURI: '', tableConfigURI: '', tableName: myTable} tlsSpec: null Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner Creating an executor service with 1 threads(Job parallelism: 0, available cores: 40.) Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS Submitting one Segment Generation Task for file:/home/sas/apache-pinot-incubating-0.7.1-bin/examples/batch/jsontype/rawdata/test.json Initialized FunctionRegistry with 119 functions: [fromepochminutesbucket, arrayunionint, codepoint, mod, sha256, year, yearofweek, upper, arraycontainsstring, arraydistinctstring, bytestohex, tojsonmapstr, trim, timezoneminute, sqrt, togeometry, normalize, fromepochdays, arraydistinctint, exp, jsonpathlong, yow, toepochhoursrounded, lower, toutf8, concat, ceil, todatetime, jsonpathstring, substr, dayofyear, contains, jsonpatharray, arrayindexofint, fromepochhoursbucket, arrayindexofstring, minus, arrayunionstring, toepochhours, toepochdaysrounded, millisecond, fromepochhours, arrayreversestring, dow, doy, min, toepochsecondsrounded, strpos, jsonpath, tosphericalgeography, fromepochsecondsbucket, max, reverse, hammingdistance, stpoint, abs, timezonehour, toepochseconds, arrayconcatint, quarter, md5, ln, toepochminutes, arraysortstring, replace, strrpos, jsonpathdouble, stastext, second, arraysortint, split, fromepochdaysbucket, lpad, day, toepochminutesrounded, fromdatetime, fromepochseconds, arrayconcatstring, base64encode, ltrim, arraysliceint, chr, sha, plus, base64decode, month, arraycontainsint, toepochminutesbucket, startswith, week, jsonformat, sha512, arrayslicestring, fromepochminutes, remove, dayofmonth, times, hour, rpad, arrayremovestring, now, divide, bigdecimaltobytes, floor, toepochsecondsbucket, toepochdaysbucket, hextobytes, rtrim, length, toepochhoursbucket, bytestobigdecimal, toepochdays, arrayreverseint, datetrunc, minute, round, dayofweek, arrayremoveint, weekofyear] in 942ms Using class: org.apache.pinot.plugin.inputformat.json.JSONRecordReader to read segment, ignoring configured file format: AVRO Finished building StatsCollector! Collected stats for 4 documents Using fixed length dictionary for column: subjects_grade, size: 20 Created dictionary for STRING column: subjects_grade with cardinality: 5, max length in bytes: 4, range: A to B-- Using fixed length dictionary for column: subjects_name, size: 5 Created dictionary for STRING column: subjects_name with cardinality: 1, max length in bytes: 5, range: maths to maths Using fixed length dictionary for column: name, size: 20 Created dictionary for STRING column: name with cardinality: 4, max length in bytes: 5, range: Pete to Pete3 Created dictionary for LONG column: age with cardinality: 4, range: 23 to 26 Start building IndexCreator! Finished records indexing in IndexCreator! Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS Start pushing segments: []... to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@4e31276e] for table myTable```
  @ken: Doesn’t the `Start pushing segments: []... to locations: ` indicate that no segments were created? Also, what’s the column that’s being used for the `dateTimeFieldSpecs`?
@mayanks: The log seems to indicate that segment was created. Perhaps it is segment push that has an issue Can you check output dir to see if there are segments there?
@ackermanholden: @ackermanholden has joined the channel
@ganesh.github: @ganesh.github has joined the channel
@jaydesai.jd: Had a question regarding “Checks” that run automatically when we raise a PR against the `apache:master` branch. I raised a PR a couple of days back and the “Checks” are not running for me. Do I need to enable some settings ? PR Reference :
  @fx19880617: you need to resolve the conflict

#pinot-dev

@badri.srinivasan: @badri.srinivasan has joined the channel

#announcements

@badri.srinivasan: @badri.srinivasan has joined the channel

#getting-started

@robersegsal: @robersegsal has joined the channel

#minion-improvements

@npawar: I’ve created this PR: @laxman
@npawar: that should help with the OOM. Would love your help in testing it out on a real setup, if you have a test environment
@laxman: surely Neha. Will do it. We are on 0.7.1. Also, we use some custom plugins. ~Is it feasible to cherry-pick this to 0.7.1 branch and provide an image?~ @fx19880617 helped me in past to provide image for a PR.
@laxman: > Is it feasible to cherry-pick this to 0.7.1 branch and provide an image? nm. I think I can do it. Will ask for help if needed. Pls give me some time to test this.
@laxman: Thanks Neha.

#fix-numerical-predicate

@irakov: @irakov has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org