yupeng9 commented on a change in pull request #6567:
URL: https://github.com/apache/incubator-pinot/pull/6567#discussion_r574013285
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/ZKOperator.java
##########
@@ -61,12 +61,15 @@ public void completeSegmentOperations(String rawTableName,
SegmentMetadata segme
URI finalSegmentLocationURI, File currentSegmentLocation, boolean
enableParallelPushProtection,
HttpHeaders headers, String zkDownloadURI, boolean
moveSegmentToFinalLocation, String crypter)
throws Exception {
- String offlineTableName =
TableNameBuilder.OFFLINE.tableNameWithType(rawTableName);
+ String tableNameWithType;
String segmentName = segmentMetadata.getName();
- // Brand new segment, not refresh, directly add the segment
- ZNRecord segmentMetadataZnRecord =
-
_pinotHelixResourceManager.getSegmentMetadataZnRecord(offlineTableName,
segmentName);
+ if (_pinotHelixResourceManager.isRealtimeOnlyTable(rawTableName)) {
+ tableNameWithType =
TableNameBuilder.REALTIME.tableNameWithType(rawTableName);
+ } else {
+ tableNameWithType=
TableNameBuilder.OFFLINE.tableNameWithType(rawTableName);
+ }
+ ZNRecord segmentMetadataZnRecord =
_pinotHelixResourceManager.getSegmentMetadataZnRecord(tableNameWithType,
segmentName);
if (segmentMetadataZnRecord == null) {
LOGGER.info("Adding new segment {} from table {}", segmentName,
rawTableName);
Review comment:
log with `tableNameWithType `?
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
##########
@@ -1649,63 +1649,81 @@ public void addNewSegment(String tableName,
SegmentMetadata segmentMetadata, Str
public void addNewSegment(String tableName, SegmentMetadata segmentMetadata,
String downloadUrl,
@Nullable String crypter) {
String segmentName = segmentMetadata.getName();
- String offlineTableName =
TableNameBuilder.OFFLINE.tableNameWithType(tableName);
-
+ String tableNameWithType;
+ InstancePartitionsType instancePartitionsType;
// NOTE: must first set the segment ZK metadata before assigning segment
to instances because segment assignment
// might need them to determine the partition of the segment, and server
will need them to download the segment
OfflineSegmentZKMetadata offlineSegmentZKMetadata = new
OfflineSegmentZKMetadata();
Review comment:
shall this be moved into the branch of `offlineTable` handling?
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentUploadDownloadRestletResource.java
##########
@@ -245,21 +245,28 @@ private SuccessResponse uploadSegment(@Nullable String
tableName, FormDataMultiP
LOGGER.info("Uploading a segment {} to table: {}, push type {},
(Derived from segment metadata)", segmentName, tableName, uploadType);
}
- String offlineTableName =
TableNameBuilder.OFFLINE.tableNameWithType(rawTableName);
+ String tableNameWithType;
+ if (_pinotHelixResourceManager.isRealtimeOnlyTable(rawTableName)) {
+ tableNameWithType =
TableNameBuilder.REALTIME.tableNameWithType(rawTableName);
+ } else {
+ tableNameWithType =
TableNameBuilder.OFFLINE.tableNameWithType(rawTableName);
+ }
String clientAddress =
InetAddress.getByName(request.getRemoteAddr()).getHostName();
LOGGER.info("Processing upload request for segment: {} of table: {} from
client: {}, ingestion descriptor: {}",
- segmentName, offlineTableName, clientAddress, ingestionDescriptor);
+ segmentName, tableNameWithType, clientAddress, ingestionDescriptor);
- // Skip segment validation if upload only segment metadata
- if (uploadType != FileUploadDownloadClient.FileUploadType.METADATA) {
+ // Skip segment validation if upload only segment metadata or it is a
realtime table segment.
+ // TODO Perform a validation check for realtime segments too.
Review comment:
Shall we allow the realtime segment upload for the upsert table only?
This can avoid the issues that @mcvsubbu pointed out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
