moszutij opened a new issue #8116:
URL: https://github.com/apache/pinot/issues/8116
Steps to reproduce
1. Setup controller with an S3 backed segment store
2. Create a segment using LaunchDataIngestionJob with a job type of
`SegmentCreationAndMetadataPush`
The segment metadata will look something like the following:
```
{
"id": "actions",
"simpleFields": {
"segment.crc": "2803120123",
"segment.creation.time": "1643896475639",
"segment.index.version": "v3",
"segment.name": "actions",
"segment.offline.download.url":
"s3://<reacted>/bi-analytics/pinot/deepstorage/actions/actions.tar.gz",
"segment.offline.push.time": "1643896476013",
"segment.table.name": "actions",
"segment.total.docs": "12",
"segment.type": "OFFLINE"
},
"mapFields": {
"custom.map": {
"input.data.file.uri":
"file:/<reacted>/bi-analytics/input/actions/2021/01/01/actions.json"
}
},
"listFields": {}
}
```
From the UI, navigate to the newly created segment and delete it. The
expected outcome is the segment file
`s3://<reacted>/bi-analytics/pinot/deepstorage/actions/actions.tar.gz` is
removed (moved to Deleted_Segments folder) from the segment store and the
segment metadata is removed from Zookeeper.
The actual outcome is the segment metadata is removed from Zookeeper and the
SegmentDeletionManager ["Fails to find local segment file for
segment"](https://github.com/apache/pinot/blob/4561b343529da8433e8ff132e0cc2a324acfda56/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java#L212).
This is because the
`fileToDeleteURI`[variable](https://github.com/apache/pinot/blob/4561b343529da8433e8ff132e0cc2a324acfda56/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java#L181)
contains `s3://<reacted>/bi-analytics/pinot/deepstorage/actions`, which
S3PinotFS.exists(...) returns as false.
See screen shot of debug session for further details:
The issue also manifests itself when data retention on a table is
configured, that is, the table's segment are not removed from the S3 segment
store.
Segment deletion works fine when the segment store is backed by a regular
file system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
