<h3><u>#general</u></h3><br><strong>@npawar: </strong>Hi all, We’re planning to
add a feature to move segments from realtime table to offline table. The
motivation is to eliminate the need for user to setup their own offline flows
for a hybrid table, and let Pinot manage that.
Here’s the design doc, called Pinot managed offline flows, if anyone wants to
take a look :
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMc9VK8AZw4xfCWnhVjqO8F0jpwxWv4fC4LAZTjvhd54Mnnp7A4BBhAtbRr8NR9LakFgJLRyTGnwArJQe4yssb40dhFrQ9yz-2FdfNludwvq-2FFP-2FKCFpQ-2BL7BgQO4LK98rpmA-3D-3DIKYA_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzaKAqb5Z6Co9dWKPN0Er2JeQebt005OYjXZym6K9A4ptaRAbRRTHeA6HnmJ1FLebnrL6AV-2BnFY9jhsZydYKCwisgxpxvJyuHkOh7xiiBp4r77J-2F-2BVy6xJdIfg-2Fgz-2F8BTGxrUdwdI0Kn1hr7-2BjFpAaV4SNRS6BBOZRGexNjztHlyyaR3riszfoDJ9gjiHFiAXY-3D><br><strong>@adrian.f.cole:
</strong>aloha. I was wondering what the client upgrade policy would be for
org.apache.pinot:pinot-java-client and org.apache.pinot:pinot-tools wrt
order<br><h3><u>#troubleshooting</u></h3><br><strong>@laxman: </strong>Hey
Folks.
Facing some issues while trying to enable pinot controller HA. Earlier,
multiple controllers are there but they were using local directory. Now, we
changed this to _gcs_ path.
After this, we notice segments are getting uploaded without any issues.
However, we see errors while trying to download these segments from gcs on
demand.<br><strong>@mayanks: </strong>what's the segment downoad url you see in
the segmentZkMetadata?<br><strong>@laxman: </strong>I see http
url<br><strong>@laxman: </strong>Here is the error
<br><strong>@laxman: </strong>```2020/08/17 15:49:55.661 ERROR
[SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel]
[HelixTaskExecutor-message_handle_thread] Caught exception in state transition
from OFFLINE -> ONLINE for resource: rawServiceView_REALTIME, partition:
rawServiceView__3__31__20200815T1540Z
java.lang.RuntimeException:
org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed
after 3 attempts
at
org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadAndReplaceSegment(RealtimeTableDataManager.java:285)
~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:251)
~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:132)
~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164)
[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_265]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_265]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404)
[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331)
[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)
[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_265]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_265]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException:
Operation failed after 3 attempts
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:61)
~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.pinot.common.utils.fetcher.HttpSegmentFetcher.fetchSegmentToLocal(HttpSegmentFetcher.java:40)
~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:108)
~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:116)
~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
at
org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadAndReplaceSegment(RealtimeTableDataManager.java:277)
~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-889889e2020f0fcbd2ef316b7fd7fe3eb985c65a]
... 14 more```<br><strong>@mayanks: </strong>Does it look like valid
gcs path?<br><strong>@mailtobuchi: </strong>@mailtobuchi has joined the
channel<br><strong>@laxman: </strong>Segment metadata for above segment
```{
"segment.realtime.endOffset": "410508",
"segment.time.unit": "MILLISECONDS",
"segment.start.time": "1597505986487",
"segment.flush.threshold.size": "13037",
"segment.realtime.startOffset": "398812",
"segment.end.time": "1597527578602",
"segment.total.docs": "11696",
"segment.table.name": "rawServiceView_REALTIME",
"segment.realtime.numReplicas": "1",
"segment.creation.time": "1597506020322",
"segment.realtime.download.url":
"<https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYatTHeqKcLaMSt8ep4ihF7S15aoAmeINWhSdVw7SqjxLtG0H3i4f81kx0fvQBcFsCGtWZbggomj6M8u1RksO5-2B3Ep5t7tJhBD685RnzcqYNPMwpfGXObjKZwxKk3nGaK4Cc93dMdtjw6oxcihNIwHyj7YwY-2BIhzYnxBDYc3FwdlaFVjTxYBuPeBvVz9KU5wOVA-3D-3D4Ua4_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzaKAqb5Z6Co9dWKPN0Er2J7a5gr43OW1CfB8lO6jNUDzs1JTDvkOzPKvwN5g7z4rKnjaivqTUaVwNsR-2BVMsFhm0-2B5u5h-2BwZQcxXY-2FFeFYI8kF9f9y7glDHtEao2LuNcN-2Fpu05Oiy554WC7TTFRo-2FpjNCww9VVwoQljlrczL7LeaDs1TBPaO81aTzpZffb-2FTbI-3D>",
"segment.name": "rawServiceView__3__31__20200815T1540Z",
"segment.index.version": "v3",
"custom.map": null,
"segment.flush.threshold.time": null,
"segment.type": "REALTIME",
"segment.crc": "2161937309",
"segment.realtime.status": "DONE"
}```<br><strong>@mayanks: </strong>Hmm, it should be gcs
path<br><strong>@laxman: </strong>No @mayanks. Its http
path.<br><strong>@laxman: </strong>Even current segments that getting prepared
its http path only…<br><strong>@mayanks: </strong>Then how do you know you are
using gcs?<br><strong>@laxman: </strong>We see data in gcs<br><strong>@laxman:
</strong>Is this related to this ticket? I see this is still open
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XKOMtdHO0f8MgDJj44LL4s9g-3D-3DhNDV_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzaKAqb5Z6Co9dWKPN0Er2JKHLcnK78-2FtpEYa7fgEr08VD-2BCN0qmCb9GzIcd7l25jw4E90HFhePiSmHSp-2Fm2R5g-2FAZvH86p7y3H4JzHtyb-2FKUkbsPP5i-2FzeMcavA6PY59Fj6Opm5bI9jwNfIHobqI0AflXV5Fs4o8EKug-2Bkx1xj6bLdUNxJMj-2FOeHliGzaE9rQ-3D><br><strong>@mayanks:
</strong>I thought I saw a fix from @kharekartik for S3 on
this?<br><strong>@g.kishore: </strong>Ting fixed it<br><strong>@laxman:
</strong>But issue is still open. Must be a duplicate.
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XKOMtdHO0f8MgDJj44LL4s9g-3D-3DNSJz_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzaKAqb5Z6Co9dWKPN0Er2J3ZMKGCEt7bB-2BCFVFfBlBBNU9CTJ5rEaiH-2B3YcLV2BsYnAaz1ubfpVgU1TcD4YBQzLxVjY-2Bfm7ZmfTqFsBeBEdOcgNZbYHsRK38DtAcpW8c26-2BWS-2BdOCu-2By-2BGw1-2FE-2BIioCewiSHsNKieou5Pxilq2ekjioKY6f20WQdd7mUZwA-2BU-3D><br><strong>@g.kishore:
</strong>i think he fixed it as part of another PR<br><strong>@g.kishore:
</strong>@tingchen ^^<br><strong>@tingchen: </strong>@tingchen has joined the
channel<br><strong>@laxman: </strong>ok. Can you please point me to the PR or
version where this is fixed?<br><strong>@laxman: </strong>Also, what happens to
the metadata for the segments when gcs was not enabled? Do we have a tool to
fix the existing metadata?<br><strong>@g.kishore: </strong>no, but its a good
idea to write one, should be simple. Do you want to give it a shot, we can
guide you<br><strong>@laxman: </strong>Sure Kishore. I’m already trying it for
a segment by manually changing the segment metadata in zookeeper using
zk-cli.<br><strong>@laxman: </strong>Meanwhile, Can you please point me to the
PR or version where this is fixed?<br><strong>@laxman: </strong>We are
currently on 0.3.0 version<br><strong>@g.kishore: </strong>the fix should be
available in 0.5.0 (going out this week)<br><strong>@elon.azoulay:
</strong>@laxman - in your controller config do you have the following set?
```<https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYW-2BdYW0ptchY-2BP-2BLfi4LyHkcaxbuXKFIoiXYEsYVXcpYYBf-2BdwiZT0qqw2z3EQg9zQCFGWO-2FMC1IqCNwCAbY5rScdCvddZ6DBJX1hLX79Jl3uhJwuJu5xIQ3OGpmhA7oaQ-3D-3Dqs3L_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzaKAqb5Z6Co9dWKPN0Er2J3SsyEYl0IL599vr3AlA4TghtCURNFS-2Bnji3OLH-2B7s2Uzwic6aYGm3X5j5JfbJi3Z388orIhyyj8DTF9aWlbGaxF0ywxPXkVA3mOjFnPNZ-2FocOjn-2FJIFyTQshjy5CIfHCHN-2BCYlHiAanFdqxW2nn2LVtO-2FAMEqXboG59SQiCJzj8-3D>=org.apache.pinot.plugin.filesystem.GcsPinotFS
pinot.controller.storage.factory.gs.projectId=<YOUR PROJECT ID>
pinot.controller.storage.factory.gs.gcpKey=<GCS KEY>
pinot.controller.segment.fetcher.protocols=file,http,gs
pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher```<br><strong>@elon.azoulay:
</strong>We had ha controllers (3 of them) in pinot 0.3.0 and 0.4.0 and this
worked for us.<br><strong>@laxman: </strong>Segment download flow from gcs
doesn’t work
Here is the ticket:
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XK4-2BMLL5cZIm1dZPgaxnPiQw-3D-3DaDnt_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzaKAqb5Z6Co9dWKPN0Er2JfltSv42HdYE6B20yGX88ygO0DVWT9arm83Mkk5sE1v-2BfwHZZLQWkQToBBLyvsE9OIJ25ZRuHRsUSsLDaNQVNWNNIhbo73DA2W-2BJgGj1ncLS8IPofsvRugi-2Bd9t3gNByRyzlOjHoX8i2Fjp7G-2B99pY01Bmi2lUR-2BWNHWm7rx77Hw-3D><br><strong>@elon.azoulay:
</strong>qq - for offline segment generation: In SegmentGeneratorConfig should
either autoGeneratedInvertedIndex or createInvertedIndexDuringSegmentGeneration
be set to true for inverted indexes to get generated?<br><strong>@elon.azoulay:
</strong>And another question for offline segment generation - just want to
confirm that sortedIndex is not used by offline, we have to provide the sorted
data, is that right?<br><strong>@christian: </strong>I’m running into an issue
with the Broker in an ERROR state for Offline tables that are backfilled using
the `SegmentCreationAndTarPush` Spark Ingestion job. When I query the hybrid
table for the min of the table it only returns the min of the realtime tables
and not the OFFLINE segments. When I query `localhost:9500/segments/TABLE_NAME`
it show both OFFLINE and ONLINE segments. I’m looking for guidance on where to
look to start debugging OFFLINE tables in `"ERROR"` an error state from the
broker’s perspective. We are using S3FileSystem.
```"sys_oops_OFFLINE": {
"CURRENT_STATE": "ERROR",
"END_TIME": "1597271220022",
"INFO": "",
"PREVIOUS_STATE": "OFFLINE",
"START_TIME": "1597271219800",
"TRIGGERED_BY":
"pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local_9000"
},
"sys_reboot_OFFLINE": {
"CURRENT_STATE": "ERROR",
"END_TIME": "1597271219935",
"INFO": "",
"PREVIOUS_STATE": "OFFLINE",
"START_TIME": "1597271219798",
"TRIGGERED_BY":
"pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local_9000"
}```<br><strong>@g.kishore: </strong>What’s in ERROR
state?<br><strong>@christian: </strong><br><strong>@christian: </strong>When I
look at `ERRORS` within the Broker Instance it’s an empty
`{}`<br><strong>@g.kishore: </strong>You should see something in that broker
log <br><strong>@g.kishore: </strong>Can you check?<br><strong>@christian:
</strong>should I just delete the offline table and recreate it
again?<br><strong>@christian:
</strong>```[HelixTaskExecutor-message_handle_thread] Caught exception while
processing transition from OFFLINE to │
│ java.lang.IllegalStateException: Failed to find schema for table:
DnsForwarderServiceStatus_OFFLINE
│
│ at
shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd
│
│ at
org.apache.pinot.broker.routing.timeboundary.TimeBoundaryManager.<init>(TimeBoundaryManager.java:70)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc │
│ at
org.apache.pinot.broker.routing.RoutingManager.buildRouting(RoutingManager.java:332)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af04
│
│ at
org.apache.pinot.broker.broker.helix.BrokerResourceOnlineOfflineStateModelFactory$BrokerResourceOnlineOfflineStateModel.onBecomeOnlineFromOffline(BrokerResourceOnlineOfflin
│
│ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_262]
│
│ at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_262]
│
│ at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_262]
│
│ at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262]
│
│ at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPS │
│ at
org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5. │
│ at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d
│
│ at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-0fc081157c88734af0433c3cd30f310474d
│
│ at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_262]
│
│ at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_262]
│
│ at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_262]
│
│ at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_262]```<br><strong>@christian: </strong>Still getting the same error
after restarting, the schema is the same as the realtime table
schema<br><strong>@g.kishore: </strong>Do you have a hen for the
table?<br><strong>@g.kishore: </strong>Sorry :neutral_face:
<br><strong>@g.kishore: </strong>Schema <br><strong>@christian: </strong>yup,
the schema is used in the realtime table so I know it
exists<br><strong>@g.kishore: </strong>Paste the offline table
config?<br><strong>@christian: </strong>```{
"OFFLINE": {
"tableName": "DnsForwarderServiceStatus_OFFLINE",
"tableType": "OFFLINE",
"segmentsConfig": {
"timeType": "MILLISECONDS",
"schemaName": "olap_enriched_dns_forwarder_service_status",
"timeColumnName": "cloud_timestamp_ms",
"replication": "1",
"replicasPerPartition": "1",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"tableIndexConfig": {
"createInvertedIndexDuringSegmentGeneration": false,
"noDictionaryColumns": [],
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": true,
"nullHandlingEnabled": true,
"loadMode": "MMAP",
"invertedIndexColumns": [
],
"autoGeneratedInvertedIndex": false
},
"metadata": {
"customConfigs": {}
}
},
"REALTIME": {
"tableName": "DnsForwarderServiceStatus_REALTIME",
"tableType": "REALTIME",
"segmentsConfig": {
"timeType": "MILLISECONDS",
"schemaName": "olap_enriched_dns_forwarder_service_status",
"timeColumnName": "cloud_timestamp_ms",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "30",
"replication": "1",
"replicasPerPartition": "1",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"tableIndexConfig": {
"createInvertedIndexDuringSegmentGeneration": false,
"noDictionaryColumns": [],
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": true,
"nullHandlingEnabled": true,
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.topic.name":
"olap-enriched-dns-forwarder-service-status",
"stream.kafka.decoder.class.name":
"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name":
"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "KAFKA_URL",
"realtime.segment.flush.threshold.time": "30m",
"realtime.segment.flush.threshold.size": "100000",
"stream.kafka.consumer.prop.auto.offset.reset": "largest",
"stream.kafka.zk.broker.url": "ZK_URL"
},
"loadMode": "MMAP",
"invertedIndexColumns": [
],
"autoGeneratedInvertedIndex": false
},
"metadata": {
"customConfigs": {}
}
}
}```<br><strong>@g.kishore: </strong>Looks right to me<br><strong>@christian:
</strong>hmm<br><strong>@npawar: </strong>i think there was some recent change
where schema name should match the table name.<br><strong>@christian:
</strong><br><strong>@npawar: </strong>```Schema schema =
ZKMetadataProvider.getTableSchema(_propertyStore, _offlineTableName);
Preconditions.checkState(schema != null, "Failed to find schema for table:
%s", _offlineTableName);```
this is throwing the exception. it needs schema with name =
DnsForwarderServiceStatus<br><strong>@christian: </strong>ah! thank you so
much!<br><strong>@npawar: </strong>upload same schema with
DnsForwarderServiceStatus name, and also, update the schema name in
tableConfigs<br><strong>@christian: </strong>that fixed
it!<br><strong>@christian: </strong>thanks so
much<br><h3><u>#release-certifier</u></h3><br><strong>@mailtobuchi:
</strong>@mailtobuchi has joined the
channel<br><h3><u>#pinot-0-5-0-release</u></h3><br><strong>@tingchen:
</strong>I got denied when trying to commit to the pinot
origin.<br><strong>@tingchen: </strong>```tingchen-C02WW14QHTDG:incubator-pinot
tingchen$ git push --set-upstream origin release-0.5.0-rc1
remote: Permission to apache/incubator-pinot.git denied to chenboat.
fatal: unable to access
'<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XKz3ZRdyLTRwG-2BZEDSfIF9jA-3D-3DVtI9_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzaKAqb5Z6Co9dWKPN0Er2J7whtgxhAqLA5U09isxVGYaJ6tEwHEFWlMpM-2BEzWc2cjNCiwkBphENQt0nTiiDxAWVkmwDhNJoLl1LmezcNWbdvQA6FWWy6P-2BrfGcPVU6pqyRr-2FwChf4gcIMV6Dne86-2B6ohUI0dx4b41NE5RJOfo9DMlg6isPA4JRVR1m9zBfn10-3D>':
The requested URL returned error: 403```
<br><strong>@tingchen: </strong>I have linked my github account chenboat to the
apache id
<mailto:[email protected]|[email protected]>.<br><strong>@tingchen:
</strong>Is the error due to the committer list not refreshed or I enter the
wrong login auth info?<br><strong>@tingchen: </strong>@npawar @g.kishore
@fx19880617<br><strong>@fx19880617: </strong>do you have write permission on
the github repo?<br><strong>@g.kishore: </strong>have you added your github id
on
<https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYRe4D53SBLSrbBa0DjRPxsaAw4MSKofH-2BH56HLrUGu5WwukF_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzaKAqb5Z6Co9dWKPN0Er2JCDo1yc1ZD4hiwlWqBv2bCPZQnrQaxLPqAIKRZOVbjtMOfS4BzZbO-2BcUPfYZcYNnNBtLFS1EjDRWwFxywdkDH-2F3PiBa7x1wKN7Ebw9Ga3zBG0xDy5semPk9JldaPahT-2BKFXskwCQzsYV44DHHuawchTpSpQ2ukr4wZYdL0-2Bttd8E-3D>?<br><h3><u>#multiple_streams</u></h3><br><strong>@fra.costa:
</strong>@g.kishore I have a question around the input connector for Realtime
tables.
Kafka has a contract of `at least once` how does Pinot preserves reporting
integrity in case Kafka outputs the same element twice into the datastore?
Thanks in advance<br><strong>@g.kishore: </strong>Kafka does not output into
Pinot <br><strong>@g.kishore: </strong>Pinot pulls from Kafka and we ensure
that every event is consumed only once<br><strong>@g.kishore: </strong>@npawar
talk at Kafka Summit will talk about this in detail <br><strong>@npawar:
</strong>@npawar has joined the channel<br><strong>@fra.costa: </strong>I
understand, great to hear, so we have one less thing to worry about
:slightly_smiling_face:<br><strong>@g.kishore: </strong>yes. we ensure
consistency across replicas as well.<br>
