#general
@jain.chitrang: @jain.chitrang has joined the channel
@gxm.monica: Hi everyone, I found if one column doesn't have text index, and I use `TEXT_MATCH` function on it as predicate _expression_, like this: ```select * from transcript where TEXT_MATCH(firstName, 'firstName*') limit 10``` it will throw exception like this: ```[ { "message": "QueryExecutionError:\njava.lang.NullPointerException\n\tat org.apache.pinot.core.operator.filter.TextMatchFilterOperator.getNextBlock(TextMatchFilterOperator.java:45)\n\tat org.apache.pinot.core.operator.filter.TextMatchFilterOperator.getNextBlock(TextMatchFilterOperator.java:30)\n\tat org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49)\n\tat org.apache.pinot.core.operator.DocIdSetOperator.getNextBlock(DocIdSetOperator.java:62)", "errorCode": 200 }, { "message": "QueryExecutionError:\njava.lang.NullPointerException\n\tat org.apache.pinot.core.operator.filter.TextMatchFilterOperator.getNextBlock(TextMatchFilterOperator.java:45)\n\tat org.apache.pinot.core.operator.filter.TextMatchFilterOperator.getNextBlock(TextMatchFilterOperator.java:30)\n\tat org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:49)\n\tat org.apache.pinot.core.operator.DocIdSetOperator.getNextBlock(DocIdSetOperator.java:62)", "errorCode": 200 } ]``` Is `TEXT_MATCH` function only used by text index columns? Like Presto, if they can't predicate expressions to connectors, they will add filter operator on top of it.So maybe it's better for users to use pql if pinot supports this syntax too?
@kennybastani: Here is the latest code for that class from the current snapshot. For reference.
@kennybastani: It's possible that `_textIndexReader` is null here, which is why that exception is getting thrown. I would open up a new issue and explain that this needs better exception handling and error reporting.
@g.kishore: @gxm.monica you can use TEXT_MATCH udf only on columns that have textIndexEnabled on other columns you can use like or regexp_like
@andrzej.holowko: @andrzej.holowko has joined the channel
@luisfernandez: hey friends, i’m using the grafana dashboard that is provided in the
@mayanks: Broker has to wait for all servers to complete, it reports the overall latency, so this is normal
@luisfernandez: thank you!!
@mayanks: Thank you for using Pinot to power use cases that require p99 is millis range :heart:
@bagi.priyank: Luis, I hope you will share your setup for achieving low latency. :slightly_smiling_face:
@luisfernandez: sure thing Priyank!
@mayanks: Thank you for using Pinot to power use cases that require p99 is millis range :heart:
@haitao: @haitao has joined the channel
#random
@jain.chitrang: @jain.chitrang has joined the channel
@andrzej.holowko: @andrzej.holowko has joined the channel
@haitao: @haitao has joined the channel
#troubleshooting
@jain.chitrang: @jain.chitrang has joined the channel
@andrzej.holowko: @andrzej.holowko has joined the channel
@jmeyer: Hello Pinot community, long time no see :slightly_smiling_face: Is there any way for `SUM` to not return 0 when there's actually no values to aggregate ? i.e. return `null` in such case
@mayanks: Long time no see @jmeyer. Do you mean no rows selected, or rows selected are null?
@jmeyer: Good question I was thinking when there's no row selected
@mayanks: Then the result should be empty iirc
@jmeyer: I remember asking this same question (couldn't find it back) and someone answered with a link to Pinot docs showing that aggregations have some kind of default values
@jmeyer: From my testing it returns 0
@jmeyer:
@mayanks: Yeah that seems to be the case, which is different from what I recall. @jackie.jxt didn’t we use to return empty results in such cases?
@jackie.jxt: No, for each aggregation function, there is a default value if no record is selected
@jmeyer: Yeah that's what I remembered, thanks @jackie.jxt Mind sharing the associated docs ? Couldn't find it back :no_mouth:
@jackie.jxt: Seems the doc has been updated and I cannot find it either :sweat_smile:
@jackie.jxt: Let me update the doc to include them
@jmeyer: I was sure it already existed lol
@jackie.jxt: @jmeyer
@jmeyer: Perfect Thanks @jackie.jxt !
@jmeyer: Thanks @mayanks too Have a good day
@mayanks: I think it is the agrr+group by that returns empty results if no records selected.
@haitao: @haitao has joined the channel
@tony: I have a realtime table with ``` "realtime.segment.flush.threshold.rows": "10000000", "realtime.segment.flush.threshold.time": "6h", "realtime.segment.flush.threshold.segment.size": "400M",``` I changed these values two days ago, previously the "rows" limit was 0. Pinot is generating segments with 3,333,333 rows, every ~90 minutes, 95-100MB -- significantly below any of the limits. Server logs show `Starting consumption on realtime consuming segment ... maxRowCount 33333` and `Stopping consumption due to row limit nRows=3333333` - I am trying to figure out where that limit is coming from.
@mayanks: `10000000` / numPartitionsOnServer
@tony: So `realtime.segment.flush.threshold.rows` is the a threshold total for all the partitions on the server - I would not have guessed that - thank you
@npawar: Also, I see you've set size threshold 400M. For that to take effect, you need to set rows threshold to 0
@tony: I assumed these were OR limits - when one limit was reached, the segment was completed. Thanks again - I am learning more.
@mayanks: cc: @mark.needham For any docs improvements
@mark.needham:
@tony: Thanks. it looks like that doc does not mention the `#rows / numPartitionsOnServer` point
@mark.needham: ah yeh good point, will update
@luisfernandez: hey friends i want to run your thoughts thru something I have been doing some chaos exercises in pinot to see how it reacts this is my current scenario: Chaos exercise in pinot: System config: 1 minion, 2 servers, 2 brokers, 3 controllers, 3 zookeepers, data replication 2, backup gcs, environment GKE Scenario: downsize to 1 server, remove server pvc, see impact, try to go back to normal. (2 servers) Steps: 1. Downsize server to 1 with kubectl scale 2. Remove pvc in server 1 with kubectl delete pvc 3. Observation: p99 response time in system still strong not noticeable changes 4. Upsize back to 2 with kubectl scale 5. Observation: things don’t kick in automatically it seems like there’s some manual steps I have to do, don’t see new server consuming and having data pulled from gcs, still see the old server in the servers UI in the pinot-controller, it seems like I need to run a rebalance at this point 6. Update offline and online tags from old server with endpoint in pinot-controller 7. Seems like we can issue a rebalance now 8. Issuing with following: dryRun=false, reassignInstances=true, includeConsuming=false, bootstrap=true, downtime=false, minAvailableReplicas=true, bestEfforts=false 9. Observation: not seeing noticeable changes in p99 response time At this point the second instance is still not in a great state and not consuming, however the system is okay performing still at ms for p99s I’m wondering the following: Question: • What to look for when a rebalance is done in the pinot-controller-logs? • When to delete the old server tag? Do I need to also issue a updateBrokerResource, I try to delete but it says that Instance Server_10.12.64.88_8098 exists in ideal state for table and it doesn’t let me drop, at this point I cannot see the tables in the UI • Any other thing I should have done while rebalancing?
@mayanks: I think there are two separate things to evaluate: ```- What happens when some nodes go down for some time and get back up (either auto or through human intervention. In this scenario, you should just disable the node (rest api), and then re-enable it. Note, no rebalance needed. - What happens if some nodes need to be physically replaced (non k8s), or add new nodes, or remove old ones. This is less of a chaos test, but more of operations test. This one requires steps on how to correctly add/remove instances and rebalance etc.```
@luisfernandez: oh I guess that i’m doing more of a operation exercise when i’m trying to go to 2 nodes again? and usually what i have been doing is scale to down and up and things usually work
@luisfernandez: tho in the server i went the extra mile of getting rid of the pvc
@luisfernandez: basically the thing that’s happening right now is that for some reason that old server is still around and the new one feels like it’s not in the rotation
@elon.azoulay: Did you try to remove the instance and it failed? You would have to rebalance the tables on it, then restart it (in the worst case), then delete the instance. Is that one of the issues you have?
@elon.azoulay: Sorry - also have to untag the server instance first
@luisfernandez: like on that particular instance? but that instance is now gone
@luisfernandez: there are 2 new ones
@luisfernandez: yep and i did the untaging
@luisfernandez: i gave it another tag basically using the update tag endpoint
@mayanks: This is a good resource on operations:
@elon.azoulay: 1 point about untagging: you have to update the instance config if you are removing tags iirc the update tags only takes non-null tags
@sandeep.hadoopadmn: Hi Team, I have problem with this timestamp, We have this date format in table "LOG_TS": "2022-03-09T16:47:42.995+00:00", and I am adding below date format, not sure if this is correct format? ``` "name": "LOG_TS", "dataType": "LONG", "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-mm-ddThh:mm:ss.sssZ", "granularity": "1:MILLISECONDS"```
@sandeep.hadoopadmn: here is table ```{ "REALTIME": { "tableName": "uapinew_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "schemaName": "uapinew", "retentionTimeUnit": "DAYS", "retentionTimeValue": "100", "timeColumnName": "LOG_TS", "allowNullTimeValue": false, "replication": "2", "replicasPerPartition": "2" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "tableIndexConfig": { "invertedIndexColumns": [], "noDictionaryColumns": [], "rangeIndexColumns": [], "rangeIndexVersion": 1, "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "sortedColumn": [], "bloomFilterColumns": [], "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "uapitranlog2", "stream.kafka.broker.list": "localhost:6667", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.segment.size": "100M" }, "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false, "aggregateMetrics": false, "nullHandlingEnabled": false }, "metadata": {}, "quota": {}, "routing": {}, "query": {}, "fieldConfigList": [ { "name": "SCHEMA_VERSION_NM", "encodingType": "RAW", "indexType": "TEXT", "indexTypes": [ "TEXT" ] } ], "ingestionConfig": {}, "isDimTable": false } }```
@sandeep.hadoopadmn: Here is Schema ```{ "schemaName": "uapinew", "dimensionFieldSpecs": [ { "name": "LOG_ID", "dataType": "STRING" }, { "name": "REQUEST_TYPE_ID", "dataType": "STRING" }, { "name": "APP_SERV_HOST_NM", "dataType": "STRING" }, { "name": "SCHEMA_VERSION_NM", "dataType": "STRING" }, { "name": "AGENCY_ID", "dataType": "STRING" }, { "name": "BRANCH_CD", "dataType": "STRING" }, { "name": "TARGET_BRANCH_CD", "dataType": "STRING" }, { "name": "PCC_CD", "dataType": "STRING" }, { "name": "AGENT_CD", "dataType": "STRING" }, { "name": "ORIGIN_APP_CD", "dataType": "STRING" }, { "name": "SUCCESS_IND", "dataType": "STRING" }, { "name": "BOOKED_SEG_QTY", "dataType": "INT" }, { "name": "AIR_SEG_BOOKED", "dataType": "STRING" }, { "name": "AIR_SEG_CANCELLED", "dataType": "STRING" }, { "name": "CAR_SEG_BOOKED", "dataType": "STRING" }, { "name": "CAR_SEG_CANCELLED", "dataType": "STRING" }, { "name": "HOTEL_SEG_BOOKED", "dataType": "STRING" }, { "name": "HOTEL_SEG_CANCELLED", "dataType": "STRING" }, { "name": "PROVIDER_CD", "dataType": "STRING" }, { "name": "SUPPLIER_CD", "dataType": "STRING" }, { "name": "PROVIDER_CD_LIST", "dataType": "STRING" }, { "name": "SUPPLIER_CD_LIST", "dataType": "STRING" }, { "name": "TRAVEL_ORIGIN_CD", "dataType": "STRING" }, { "name": "TRAVEL_DEST_CD", "dataType": "STRING" }, { "name": "TRAVEL_START_DT", "dataType": "STRING" }, { "name": "TRAVEL_END_DT", "dataType": "STRING" }, { "name": "NUM_TRAVELERS_QTY", "dataType": "STRING" }, { "name": "UR_LOCATOR_CD", "dataType": "STRING" }, { "name": "ONE_WAY_SHOP_IND", "dataType": "STRING" }, { "name": "HOST_LOCATOR_CD", "dataType": "STRING" }, { "name": "SUPPLIER_LOCATOR_CD", "dataType": "STRING" }, { "name": "OPTIONAL_SVC_QTY", "dataType": "STRING" }, { "name": "TRAVEL_SEGMENTS", "dataType": "STRING" }, { "name": "TRAVEL_DATES", "dataType": "STRING" }, { "name": "FLIGHT_NBR", "dataType": "STRING" }, { "name": "CLASS_OF_SERVICE", "dataType": "STRING" }, { "name": "AVAILABILITY_SOURCE", "dataType": "STRING" }, { "name": "NBR_IN_PARTY", "dataType": "STRING" }, { "name": "HOTEL_CODE", "dataType": "STRING" }, { "name": "HOTEL_SEARCH_BY_ADDR_IND", "dataType": "STRING" }, { "name": "SSL_TERM_REGION", "dataType": "STRING" }, { "name": "REQUEST_BASE_PRICE", "dataType": "STRING" }, { "name": "RESPONSE_BASE_PRICE", "dataType": "STRING" }, { "name": "BASE_DIFFERENCE_TOTAL", "dataType": "STRING" }, { "name": "REQUEST_TOTAL_PRICE", "dataType": "STRING" }, { "name": "RESPONSE_TOTAL_PRICE", "dataType": "STRING" }, { "name": "TOTAL_DIFFERENCE_PRICE", "dataType": "STRING" }, { "name": "SPLIT_TICKET_SEARCH", "dataType": "STRING" }, { "name": "SPLIT_TICKET_VALID", "dataType": "STRING" }, { "name": "SPLIT_TICKET_PERCENT", "dataType": "STRING" }, { "name": "GAL_RT_MILLISEC_QTY", "dataType": "STRING" }, { "name": "APO_RT_MILLISEC_QTY", "dataType": "STRING" }, { "name": "WSP_RT_MILLISEC_QTY", "dataType": "STRING" }, { "name": "JAL_RT_MILLISEC_QTY", "dataType": "STRING" }, { "name": "ACH_RT_MILLISEC_QTY", "dataType": "STRING" }, { "name": "RCH_RT_MILLISEC_QTY", "dataType": "STRING" }, { "name": "MCH_RT_MILLISEC_QTY", "dataType": "STRING" }, { "name": "TRM_RT_MILLISEC_QTY", "dataType": "STRING" }, { "name": "SVCS_RT_MILLISEC_QTY", "dataType": "STRING" }, { "name": "REQUEST_BYTES_QTY", "dataType": "STRING" }, { "name": "RESPONSE_BYTES_QTY", "dataType": "STRING" }, { "name": "ERROR_LEVEL_CD", "dataType": "STRING" }, { "name": "ERROR_CD", "dataType": "STRING" }, { "name": "ERROR_TYPE_CD", "dataType": "STRING" }, { "name": "ERROR_TEXT", "dataType": "STRING" }, { "name": "SOAP_FAULT_TEXT", "dataType": "STRING" }, { "name": "TRACE_ID", "dataType": "STRING" }, { "name": "FARE_RULES_FILTER_CAT_CODES", "dataType": "STRING" }, { "name": "REQ_RATE_PLAN", "dataType": "STRING" }, { "name": "NUM_ADULTS", "dataType": "STRING" }, { "name": "NUM_ROOMS", "dataType": "STRING" }, { "name": "GUARANTEE_TYPE", "dataType": "STRING" }, { "name": "OTHER_GUARANTEE_INFO", "dataType": "STRING" }, { "name": "RSP_RATE_PLAN", "dataType": "STRING" }, { "name": "RESPONSE_MSG_CODES", "dataType": "STRING" }, { "name": "HOTEL_CONF_NUM", "dataType": "STRING" }, { "name": "HOTEL_STATUS", "dataType": "STRING" }, { "name": "LANGUAGE_CODE", "dataType": "STRING" }, { "name": "PICKUP_LOC_TYPE", "dataType": "STRING" }, { "name": "RETURN_LOC_TYPE", "dataType": "STRING" }, { "name": "PICKUP_LOC_NUM", "dataType": "STRING" }, { "name": "RETURN_LOC_NUM", "dataType": "STRING" }, { "name": "VEH_RATE_CODE", "dataType": "STRING" }, { "name": "VEH_RATE_CAT", "dataType": "STRING" }, { "name": "VEH_RATE_PERIOD", "dataType": "STRING" }, { "name": "VEH_DISCOUNT_NUM", "dataType": "STRING" }, { "name": "ACRISS_VEH_CODE", "dataType": "STRING" }, { "name": "VEH_RATE_FOR_PERIOD", "dataType": "STRING" }, { "name": "RSP_BASE_RATE", "dataType": "STRING" }, { "name": "CDS_TXID", "dataType": "STRING" }, { "name": "CDS_TIME", "dataType": "STRING" }, { "name": "CDS_STATUS", "dataType": "STRING" }, { "name": "CDS_ERROR_MSG", "dataType": "STRING" }, { "name": "ESTIMATED_TOTAL_AMOUNT", "dataType": "STRING" }, { "name": "MANDATORY_CHARGE_TOTAL", "dataType": "STRING" }, { "name": "AIR_CONDITIONING", "dataType": "STRING" }, { "name": "TRANSMISSION_TYPE", "dataType": "STRING" }, { "name": "VEHICLE_CLASS", "dataType": "STRING" }, { "name": "CATEGORY", "dataType": "STRING" }, { "name": "VEHICLE_TYPE", "dataType": "STRING" }, { "name": "DOOR_COUNT", "dataType": "STRING" }, { "name": "AUX1", "dataType": "STRING" }, { "name": "AUX2", "dataType": "STRING" }, { "name": "AUX3", "dataType": "STRING" }, { "name": "CANCELLATION_TEXT", "dataType": "STRING" }, { "name": "REQUESTED_NEGOTIATED_RATE_CODES_COUNT", "dataType": "STRING" }, { "name": "RATE_CATEGORY_COUNT", "dataType": "STRING" }, { "name": "NUMBER_OF_RATE_PLAN_TYPES", "dataType": "STRING" }, { "name": "NEGOTIATED_RATE_CODES_IN_RESPONSE", "dataType": "STRING" }, { "name": "RESPONSE_NEGOTIATED_RATE_CODE_COUNT", "dataType": "STRING" }, { "name": "REQUESTED_VENDOR_LOCATION_COUNT", "dataType": "STRING" }, { "name": "MAX_WAIT_TIME", "dataType": "STRING" }, { "name": "RESPONSE_PROPERTIES_COUNT", "dataType": "STRING" }, { "name": "TAX", "dataType": "STRING" }, { "name": "SURCHARGE", "dataType": "STRING" }, { "name": "COMMISION_INDICATOR", "dataType": "STRING" }, { "name": "PERCENTAGE", "dataType": "STRING" }, { "name": "GALILEO_GUARANTEE", "dataType": "STRING" }, { "name": "GALILEO_DEPOSIT", "dataType": "STRING" }, { "name": "WORLDSPAN_DEPOSIT_GUARANTEE_PREPAYMENT", "dataType": "STRING" }, { "name": "UAPI_RESPONSE_TIME_MS", "dataType": "STRING" }, { "name": "SESSION_KEY", "dataType": "STRING" }, { "name": "AGENCY_COUNTRY", "dataType": "STRING" }, { "name": "SOURCE_IP_ADDRESS", "dataType": "STRING" }, { "name": "ACH_TRANSACTION_ID", "dataType": "STRING" }, { "name": "DB2_LOCK_TIME", "dataType": "STRING" }, { "name": "DB2_NUMBER_OF_LOCKS", "dataType": "STRING" }, { "name": "OVERRIDE_PCC", "dataType": "STRING" }, { "name": "META_SEARCH", "dataType": "STRING" }, { "name": "SOLUTION_RESULT", "dataType": "STRING" }, { "name": "PTC_LIST", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "TOTAL_MILLISECOND_QTY", "dataType": "INT" } ], "dateTimeFieldSpecs": [ { "name": "LOG_TS", "dataType": "LONG", "format": "1:MILLISECONDS:SIMPLE_DATE_FORMAT:yyyy-mm-ddThh:mm:ss.sssZ", "granularity": "1:MILLISECONDS" } ] }```
@sandeep.hadoopadmn: Getting below error , "DOOR_COUNT" : "", "ESTIMATED_TOTAL_AMOUNT" : "", "DB2_NUMBER_OF_LOCKS" : "0", "TRAVEL_END_DT" : "", "ACRISS_VEH_CODE" : "", "SOLUTION_RESULT" : "", "BASE_DIFFERENCE_TOTAL" : "", "RESPONSE_BASE_PRICE" : "", "BOOKED_SEG_QTY" : "0", "REQUESTED_VENDOR_LOCATION_COUNT" : "", "CLASS_OF_SERVICE" : "", "NUM_ADULTS" : "", "SESSION_KEY" : "", "SUPPLIER_LOCATOR_CD" : "", "OTHER_GUARANTEE_INFO" : "", "SSL_TERM_REGION" : "\"216.113.156.89:443|/TF-COM_API/EMEA.universal-api.travelport.com_443\"", "REQUEST_BASE_PRICE" : "", "SOAP_FAULT_TEXT" : "", "SURCHARGE" : "" }, "nullValueFields" : [ ] } java.lang.RuntimeException: Caught exception while transforming data type for column: LOG_TS at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.transform(DataTypeTransformer.java:95) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.segment.local.recordtransformer.CompositeTransformer.transform(CompositeTransformer.java:83) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:518) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:568) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.lang.NumberFormatException: For input string: "2022-03-10T05:30:08.627+00:00" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:?] at java.lang.Long.parseLong(Long.java:692) ~[?:?] at java.lang.Long.parseLong(Long.java:817) ~[?:?] at org.apache.pinot.common.utils.PinotDataType$10.toLong(PinotDataType.java:502) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.common.utils.PinotDataType$6.convert(PinotDataType.java:333) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.common.utils.PinotDataType$6.convert(PinotDataType.java:290) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.transform(DataTypeTransformer.java:90) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] ... 5 more 2022/03/11 05:42:03.583 ERROR [LLRealtimeSegmentDataManager_uapinew__24__0__20220311T0541Z] [uapinew__24__0__20220311T0541Z] Caught exception while transforming the record: { "fieldToValueMap" : { "TRAVEL_DEST_CD" : "", "PCC_CD" : "3S0E", "SUPPLIER_CD_LIST" : "QR|SU|OS|EK|MU|OZ|LX", "RESPONSE_PROPERTIES_COUNT" : "", "ACH_TRANSACTION_ID" : "", "PROVIDER_CD" : "SVCS", "RESPONSE_BYTES_QTY" : "971876", "VEH_RATE_PERIOD" : "", "JAL_RT_MILLISEC_QTY" : "0", "AUX2" : "", "AUX3" : "", "META_SEARCH" : "", "SVCS_RT_MILLISEC_QTY" : "2797", "RSP_RATE_PLAN" : "", "AUX1" : "", "GALILEO_GUARANTEE" : "", "SCHEMA_VERSION_NM" : "SessionContext_v1|air_v45_0|rail_v45_0", "VEH_RATE_FOR_PERIOD" : "",^Z [4]+ Stopped tail -f pinot-all.log [root@vhldvatlb211 logs]# vi pinot-all.log [5]+ Stopped vi pinot-all.log [root@vhldvatlb211 logs]# vi pinot-all.log [6]+ Stopped vi pinot-all.log [root@vhldvatlb211 logs]# vi pinot-all.log at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:568) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.lang.NumberFormatException: For input string: "2022-03-10T05:43:28.983+00:00" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:?] at java.lang.Long.parseLong(Long.java:692) ~[?:?] at java.lang.Long.parseLong(Long.java:817) ~[?:?] at org.apache.pinot.common.utils.PinotDataType$10.toLong(PinotDataType.java:502) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.common.utils.PinotDataType$6.convert(PinotDataType.java:333) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.common.utils.PinotDataType$6.convert(PinotDataType.java:290) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.transform(DataTypeTransformer.java:90) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] ... 5 more 2022/03/11 05:45:00.989 ERROR [LLRealtimeSegmentDataManager_uapinew__26__0__20220311T0541Z] [uapinew__26__0__20220311T0541Z] Caught exception while transforming the record: { "fieldToValueMap" : {P_RT_MILLISEC_QTY" : "0", "RSP_BASE_RATE" : "", "CAR_SEG_BOOKED" : "0", "RESPONSE_BASE_PRICE" : "", "BOOKED_SEG_QTY" : "0", "REQUESTED_VENDOR_LOCATION_COUNT" : "", "CLASS_OF_SERVICE" : "", "NUM_ADULTS" : "", "SESSION_KEY" : "", "SUPPLIER_LOCATOR_CD" : "", "OTHER_GUARANTEE_INFO" : "", "SSL_TERM_REGION" : "\"216.113.156.89:443|/TF-COM_API/EMEA.universal-api.travelport.com_443\"", "REQUEST_BASE_PRICE" : "", "SOAP_FAULT_TEXT" : "", "SURCHARGE" : "" }, "nullValueFields" : [ ] } java.lang.RuntimeException: Caught exception while transforming data type for column: LOG_TS at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.transform(DataTypeTransformer.java:95) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.segment.local.recordtransformer.CompositeTransformer.transform(CompositeTransformer.java:83) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:518) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:568) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.lang.NumberFormatException: For input string: "2022-03-10T05:30:08.627+00:00" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:?] at java.lang.Long.parseLong(Long.java:692) ~[?:?] at java.lang.Long.parseLong(Long.java:817) ~[?:?] at org.apache.pinot.common.utils.PinotDataType$10.toLong(PinotDataType.java:502) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.common.utils.PinotDataType$6.convert(PinotDataType.java:333) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.common.utils.PinotDataType$6.convert(PinotDataType.java:290) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.transform(DataTypeTransformer.java:90) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] ... 5 more 2022/03/11 05:42:03.583 ERROR [LLRealtimeSegmentDataManager_uapinew__24__0__20220311T0541Z] [uapinew__24__0__20220311T0541Z] Caught exception while transforming the record: { "fieldToValueMap" : { "TRAVEL_DEST_CD" : "", "PCC_CD" : "3S0E", "SUPPLIER_CD_LIST" : "QR|SU|OS|EK|MU|OZ|LX", "RESPONSE_PROPERTIES_COUNT" : "", "ACH_TRANSACTION_ID" : "", "PROVIDER_CD" : "SVCS", "RESPONSE_BYTES_QTY" : "971876", "VEH_RATE_PERIOD" : "", "JAL_RT_MILLISEC_QTY" : "0", "AUX2" : "", "AUX3" : "", "META_SEARCH" : "", "SVCS_RT_MILLISEC_QTY" : "2797", "RSP_RATE_PLAN" : "", "AUX1" : "", "GALILEO_GUARANTEE" : "", "SCHEMA_VERSION_NM" : "SessionContext_v1|air_v45_0|rail_v45_0", "VEH_RATE_FOR_PERIOD" : "",^Z [4]+ Stopped tail -f pinot-all.log [root@vhldvatlb211 logs]# vi pinot-all.log [5]+ Stopped vi pinot-all.log [root@vhldvatlb211 logs]# vi pinot-all.log [6]+ Stopped vi pinot-all.log [root@vhldvatlb211 logs]# vi pinot-all.log at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:568) [pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.lang.NumberFormatException: For input string: "2022-03-10T05:43:28.983+00:00" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:?] at java.lang.Long.parseLong(Long.java:692) ~[?:?] at java.lang.Long.parseLong(Long.java:817) ~[?:?] at org.apache.pinot.common.utils.PinotDataType$10.toLong(PinotDataType.java:502) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.common.utils.PinotDataType$6.convert(PinotDataType.java:333) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.common.utils.PinotDataType$6.convert(PinotDataType.java:290) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] at org.apache.pinot.segment.local.recordtransformer.DataTypeTransformer.transform(DataTypeTransformer.java:90) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb] ... 5 more 2022/03/11 05:45:00.989 ERROR [LLRealtimeSegmentDataManager_uapinew__26__0__20220311T0541Z] [uapinew__26__0__20220311T0541Z] Caught exception while transforming the record: { "fieldToValueMap" : {
@sandeep.hadoopadmn: This is happening in 0.9.3, but the same table and schema is working on 0.8
@sandeep.hadoopadmn: @mayanks ^^
@sandeep.hadoopadmn: attached log file
#getting-started
@jain.chitrang: @jain.chitrang has joined the channel
@andrzej.holowko: @andrzej.holowko has joined the channel
@haitao: @haitao has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
