Apache Pinot Daily Email Digest (2020-06-24)

[Pinot Slack Email Digest]

<h3><u>#troubleshooting</u></h3><br><strong>@pradeepgv42: </strong>I created a 
table with following tagOverrideConfig

```  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant",
    "tagOverrideConfig": {
      "realtimeConsuming": "DefaultTenant_REALTIME",
      "realtimeCompleted": "DefaultTenant_OFFLINE"
    }
  },```
But the restapi /tables/{tableName} only shows
```"tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },```
Wondering if this config is coupled with something else?<br><strong>@g.kishore: 
</strong>@npawar ^^ any docs on this?<br><strong>@g.kishore: 
</strong><https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMdTeAXadp8BL3QinSdRtJdpjVdLs6pcP-2BmJYu0RGOJgjg3f76AVCInMqjLW2FlvR9ePnJ4sDiWaCnI3UkhMZ1H8-3DWHDt_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzA0q2WPCG8PY-2BfIL0dR3lVUqsh-2BcDQXfbGb4VvBqVWfz57eERbHTvtv-2Bje-2F3RKo2T6FWcUOoBAE2DW3gzEA03ELXm2ch6n5Z6rZCAD1M2aH0RoOgkwP2EmekvznUTyWaP4-2FkuiC0zqb1q-2BS7FmaayJtSd48uR0Zn2N81ZDDi-2FJIqng4DUcU327-2FQJ9VEAfgn4-3D>
 we should add the real-time segment relocator here<br><strong>@pradeepgv42: 
</strong>There’s some info here: 
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMdTeAXadp8BL3QinSdRtJdpzMyfof-2FnbcthTx3PKzMZIuwKP6nnHI6Bhy90QCctKQTAnI6DJEG-2BdM1AiPNpzzvs-3DjCUS_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzA0q2WPCG8PY-2BfIL0dR3lV1t740DdMg6D8yO-2FoLMxrJIo2SZkQzHtTH569Teo-2FN-2BEBRgNvqo4izS935I2PsMNCDQ-2BoeoXa4V603tUB9JSf7AFzJzZygvZw4Yt7uus8PhYK26D4LqDsID-2BnyLpRDScZto6-2FiAigaPdutMUqHV0bwVzi-2FHZk5u4x16Ckd6jc9jo-3D><br><strong>@npawar:
 </strong>looking<br><strong>@g.kishore: </strong>cool, 
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMdTeAXadp8BL3QinSdRtJdoH1omTMY3qwbAUC5IutlmIfnExuLFuaw8aCfHk5tae35jst1BsOURaWXx6d7RyTlJ7LtIrWrIV2IL80EKu0oHFjdyH5yTUODaJYNqqLVBl03ZwVDqr-2FMFhF6cc-2BTE7Jgw-3DhU2B_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzA0q2WPCG8PY-2BfIL0dR3lVV7hBCk8osl59BMKev1Fa1R588F-2FJUs5DMrXHs7fHwdHueG-2BP8PTSBGTxKRSWB3HIxwq-2FDAx-2F7XXm34-2F8we7YKe2g-2B5bScQrYCAxzYvfiq8r1Aar5IJn1ako6H-2FlwUknVstmx4uAkYg2q7fNqK1hbzByZOSOOx-2BSFXvBrPz7-2FAsg-3D><br><strong>@pradeepgv42:
 </strong>I am using “LowLevel” consumer<br><strong>@npawar: </strong>can you 
share the whole table config?<br><strong>@pradeepgv42: </strong>```{
  "REALTIME": {
    "tableName": "&lt;REMOVED&gt;",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "schemaName": "search",
      "timeType": "MILLISECONDS",
      "timeColumnName": "timestampMillis",
      "replicasPerPartition": "1"
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.consumer.type": "LowLevel",
        "stream.kafka.topic.name": "&lt;REMOVED&gt;",
        "stream.kafka.decoder.class.name": 
"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
        "stream.kafka.consumer.factory.class.name": 
"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.broker.list": "&lt;REMOVED&gt;",
        "realtime.segment.flush.threshold.time": "1h",
        "realtime.segment.flush.threshold.size": "100M",
        "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
      },
      "createInvertedIndexDuringSegmentGeneration": false,
      "enableDefaultStarTree": false,
      "aggregateMetrics": false,
      "nullHandlingEnabled": false,
      "loadMode": "MMAP",
      "autoGeneratedInvertedIndex": false
    },
    "metadata": {
      "customConfigs": {}
    }
  }
}```<br><strong>@npawar: </strong>also check if there’s anything in the logs 
when you did a POST for the table config<br><strong>@pradeepgv42: </strong>let 
me check that<br><strong>@npawar: </strong>also, was this during 1st time 
creation, or update?<br><strong>@g.kishore: </strong>btw, I don't think there 
is anything wrong with the output.<br><strong>@pradeepgv42: </strong>@npawar 
First time I had tagsOverrideConfig but consumer type was “lowlevel”, so I did 
AddTable again where I changed consumer type from lowlevel to LowLevel
Not sure if that would effect?<br><strong>@npawar: </strong>that property is 
case insensitive, so both calls should be treated the same<br><strong>@npawar: 
</strong>i could reproduce, i’ll investigate<br><strong>@pradeepgv42: 
</strong>nice thanks<br><strong>@npawar: </strong>how are you starting the 
Controller? @pradeepgv42<br><strong>@pradeepgv42: </strong>bin/pinot-admin.sh   
 StartController -configFileName  &lt;controllerConfig&gt;<br><strong>@npawar: 
</strong>cool, there’s a controller property called tenantIsolation which by 
default is true. that prop overrides the tenant config.<br><strong>@npawar: 
</strong>we can set it to false in the controller conf. looking for the exact 
property key<br><strong>@pradeepgv42: </strong>would that effect anything else? 
yeah not able to find a way to override it through controller 
config<br><strong>@npawar: 
</strong>`cluster.tenant.isolation.enable=false`<br><strong>@npawar: 
</strong>try this<br><strong>@pradeepgv42: </strong>trying 
it..<br><strong>@pradeepgv42: </strong>it worked, I can see the 
tagsOverrideConfig from the rest-api, I will wait for a bit and check if the 
segments are moving<br><strong>@pradeepgv42: </strong>What does the 
`cluster.tenant.isolation.enable` variable do?<br><strong>@pradeepgv42: 
</strong>it assumes single tenant?<br><strong>@npawar: </strong>Segments are 
moved by a periodic task on the controller. so you might have to wait upto an 
hour to see the completed segments moving. it runs every 
hour<br><strong>@npawar: </strong>yes.. that property assumes single tenant 
called “DefaultTenant”<br><strong>@g.kishore: </strong>@npawar do we still need 
that property, looks like its not adding any value<br><strong>@g.kishore: 
</strong>maybe it was added to roll out the feature in a controlled 
manner<br><strong>@npawar: </strong>actually i dont know why we have that prop. 
i can check and remove it if i dont find any use<br><strong>@g.kishore: 
</strong>cool, lets create an issue and remove it<br><strong>@pradeepgv42: 
</strong>makes sense, thanks, will check a bit later<br><strong>@npawar: 
</strong>created issue: 
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XKHGIZPmhs5qanOPzkt3uFNQ-3D-3D-Wwz_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzA0q2WPCG8PY-2BfIL0dR3lVO-2FHjOD8ld7B609-2FcZwFtVyug89FwDaB542XOGnSct5Oa0iZJQEYeFXHoyuywrbbJEjBKJLh680DFSMOQNw7GNAI47gIr-2B1DP2No4yOp2m0OjjNG58t1ACSOET-2F-2FnuAu-2FakL2rpl0XXImMuApwsQNjU3LdDL5OluunLzxY31eTxc-3D><br>