#general
@robeng962: Hi there, how do I pass taints and tolerations as params to the helm chart? Been googling the syntax but can't find correct way to do it. An example will be much appreciated thanks
@xiangfu0: you can export all values to values.yaml file, then edit it, then helm install with values.yaml file
@robeng962: Sure, I'll try that
@nizar.hejazi: Hey there, I want to write a transformation to convert a datetime column that has string values (e.g. ‘2022-03-19T11:00:18.789Z’) or nulls into timestamps. Using inbuilt function (FromDateTime) throws `java.lang.NullPointerException` when the value is null: ```{ "columnName": "updatedAt_timestamp", "transformFunction": "FromDateTime(updatedAt, 'yyyy-MM-dd''T''HH:mm:ss.SSS''Z''')" }``` Trying to use Groovy script as following but I see the following exception in the logs: `MissingPropertyException: No such property: DateTimeFormat for class: Script1` ```{ "columnName": "col_timestamp", "transformFunction": "Groovy({col == null ? null : DateTimeFormat.forPattern('yyyy-MM-dd\\'T\\'HH:mm:ss.SSS\\'Z\\'').withZone(DateTimeZone.forID(DateTimeZone.UTC.getID())).parseMillis(adminAccessGrantedOn)}, col)" },``` Do I need to import joda time classes to Groovy? Can I write a multi-line Groovy script as an ingestion transform? Any other workaround to deal w/ nulls in FromDateTime inbuilt function? (I can submit a PR to update date time functions to handle nulls). Please note that I have `"nullHandlingEnabled"` set to True.
@nizar.hejazi: Note: the following Groovy script is valid (tested in a Groovy compiler): ```@Grab( 'joda-time:joda-time:2.10.5' ) import org.joda.time.DateTimeZone import org.joda.time.format.DateTimeFormat import org.joda.time.format.DateTimeFormatter dt = '2022-03-19T11:00:18.789Z' print DateTimeFormat.forPattern('yyyy-MM-dd\'T\'HH:mm:ss.SSS\'Z\'').withZone(DateTimeZone.forID(DateTimeZone.UTC.getID())).parseMillis(dt)```
@mayanks: Groovy script cannot call Pinot defined UDFs. But you should be able to do that transform without the Groovy script?
@nizar.hejazi: Do you mean using inbuilt functions only? I cannot find inbuilt function for handling nulls (IsNull, IF, etc.)
@nizar.hejazi: I need to return null if the input column value is null, otherwise: convert the string datetime to timestamp
@mayanks:
@mayanks: Is this time column? If so, you probably don’t want to convert to null
@kharekartik: We allow it to convert to a default value (which can be set by you), in case of nulls. However, returning a null is not supported.
@nizar.hejazi: this is one of the three available time columns that I use for filtering (not as timeColumnName in segmentsConfig), maybe I should define it as dimensionSpec?
@nizar.hejazi: If I have `"nullHandlingEnabled"` set to True, why I cannot return null in an ingestion transform?
@kharekartik: The functions can't return nulls. When `nullHandlingEnabled` is set to true, whatever data you have uploaded, if the column is empty, we mark it as null in our index, the value is still stored as a default one.
@nizar.hejazi: What happens if the column is empty (null) and there is an ingestion transform defined? For example: JSONPATH takes an object and return null if the input is null. I need to do the same for FromDateTime.
@nizar.hejazi: See:
@nizar.hejazi: @kharekartik The logic is as following: • If col is null, set to whatever default value it should take and mark it as null in the index. • If col is not null, transform it from a string datetime to a timestamp. Can I achieve this one way or another?
@kharekartik: Can I ask you why you need to convert from string to timestamp?
@nizar.hejazi: for perf reasons .. wanna pay the cost at ingestion time in order for queries to run faster
@nizar.hejazi: Can I submit a PR to check for if dateTimeString is null in `FromDateTime` and `ToDateTime` functions and return null in this case? Source code:
@kharekartik: @jackie.jxt We had a discussion on this earlier and the conclusion was to raise exceptions in this case. Can we do it some other way?
@nizar.hejazi: suggestion: If `nullHandlingEnabled` is set to true, returns null. Otherwise, throw an exception.
@kharekartik: Issue is for columns with DataType other than Object, setting to null with throw NullPointerException in results.
@nizar.hejazi: @kharekartik @jackie.jxt one simple solution here is as following: If the input column value is null, check `nullHandlingEnabled` setting: • If `nullHandlingEnabled` is set to true, mark the value as null in the null index, and avoid calling the transform ingestion function. • If `nullHandlingEnabled` is set to false, call the transform ingestion function.
@diogo.baeder: Hi there folks! I've just published an article about YouGov starting to use Apache Pinot, in my personal blog - I can't yet publish it in a more appropriate YouGov public engineering blog, so I opted by having it in my own site for now. Here it is:
@mayanks: Thanks @diogo.baeder
@diogo.baeder: No problem at all; Thank *you*! :slightly_smiling_face:
@diogo.baeder: Oops, I forgot the "Thanks" section; Fixed!
@mayanks: Thanks again @diogo.baeder, I think new community members who are exploring different OLAP solutions will find this really useful.
@diogo.baeder: I hope so! :slightly_smiling_face:
@diogo.baeder: Hopefully I'm able to move this to a YouGov official blog, but I think that's good enough for now.
@diogo.baeder: @mayanks ^ as promised :slightly_smiling_face:
@mayanks: Thanks again @diogo.baeder, I think new community members who are exploring different OLAP solutions will find this really useful.
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
