xiangfu0 opened a new pull request, #10426:
URL: https://github.com/apache/pinot/pull/10426
Seeing logs like:
```
Listed 2 files from URI:
s3://my-pinot-segments/stg-bluering-pinot/controller-data/myTable/, is
recursive: true
Start pushing segment metadata:
{s3://my-pinot-segments/stg-bluering-pinot/controller-data/myTable/myTable__1__139__20230309T2250Z.tar.gz=s3://my-pinot-segments/stg-bluering-pinot/controller-data/myTable/myTable__1__139__20230309T2250Z.tar.gz,
s3://my-pinot-segments/stg-bluering-pinot/controller-data/myTable/myTable__3__146__20230311T2306Z.tar.gz=s3://my-pinot-segments/stg-bluering-pinot/controller-data/myTable/myTable__3__146__20230311T2306Z.tar.gz}
to locations:
[org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@46b6701e] for table
myTable
Copy
s3://my-pinot-segments/stg-bluering-pinot/controller-data/myTable/myTable__1__139__20230309T2250Z.tar.gz
to local /tmp/segmentTar-6b4e04df-a531-4301-bda2-4168c1279961.tar.gz
Trying to untar Metadata file from:
[/tmp/segmentTar-6b4e04df-a531-4301-bda2-4168c1279961.tar.gz] to
[/tmp/segmentMetadataDir-6b4e04df-a531-4301-bda2-4168c1279961]
Trying to untar CreationMeta file from:
[/tmp/segmentTar-6b4e04df-a531-4301-bda2-4168c1279961.tar.gz] to
[/tmp/segmentMetadataDir-6b4e04df-a531-4301-bda2-4168c1279961]
Trying to tar segment metadata dir
[/tmp/segmentMetadataDir-6b4e04df-a531-4301-bda2-4168c1279961] to
[/tmp/segmentMetadata-6b4e04df-a531-4301-bda2-4168c1279961.tar.gz]
Pushing segment: myTable__1__139__20230309T2250Z to location:
http://my-pinot-controller:9000 for table myTable
Sending request:
http://my-pinot-controller:9000/v2/segments?tableName=myTable to controller:
ip-xx-xxx-xxx-xx.my-zone.compute.internal, version: Unknown
Response for pushing table myTable segment myTable__1__139__20230309T2250Z
to location http://my-pinot-controller:9000 - 200: {"status":"Successfully
uploaded segment: myTable__1__139__20230309T2250Z of table: myTable_OFFLINE"}
Copy
s3://my-pinot-segments/stg-bluering-pinot/controller-data/myTable/myTable__3__146__20230311T2306Z.tar.gz
to local /tmp/segmentTar-9d46d451-c744-40eb-bf02-f4bd50e5262a.tar.gz
Trying to untar Metadata file from:
[/tmp/segmentTar-9d46d451-c744-40eb-bf02-f4bd50e5262a.tar.gz] to
[/tmp/segmentMetadataDir-9d46d451-c744-40eb-bf02-f4bd50e5262a]
Trying to untar CreationMeta file from:
[/tmp/segmentTar-9d46d451-c744-40eb-bf02-f4bd50e5262a.tar.gz] to
[/tmp/segmentMetadataDir-9d46d451-c744-40eb-bf02-f4bd50e5262a]
Trying to tar segment metadata dir
[/tmp/segmentMetadataDir-9d46d451-c744-40eb-bf02-f4bd50e5262a] to
[/tmp/segmentMetadata-9d46d451-c744-40eb-bf02-f4bd50e5262a.tar.gz]
Pushing segment: myTable__3__146__20230311T2306Z to location:
http://my-pinot-controller:9000 for table myTable
Sending request:
http://my-pinot-controller:9000/v2/segments?tableName=myTable to controller:
ip-xx-xxx-xxx-xx.my-zone.compute.internal, version: Unknown
Caught temporary exception while pushing table: myTable segment:
myTable__3__146__20230311T2306Z to http://my-pinot-controller:9000, will retry
org.apache.pinot.common.exception.HttpErrorStatusException: Got error status
code: 500 (Internal Server Error) with reason: "Exception while uploading
segment: No enum constant
org.apache.pinot.common.utils.FileUploadDownloadClient.FileUploadType.METADATA,METADATA"
while sending request:
http://my-pinot-controller:9000/v2/segments?tableName=myTable to controller:
ip-xx-xxx-xxx-xx.my-zone.compute.internal, version: Unknown
at
org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:442)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.common.utils.FileUploadDownloadClient.uploadSegmentMetadata(FileUploadDownloadClient.java:583)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.lambda$sendSegmentUriAndMetadata$2(SegmentPushUtils.java:314)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUriAndMetadata(SegmentPushUtils.java:304)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUriAndMetadata(SegmentPushUtils.java:136)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner.uploadSegments(SegmentMetadataPushJobRunner.java:38)
[pinot-batch-ingestion-standalone-0.12.0-shaded.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.plugin.ingestion.batch.common.BaseSegmentPushJobRunner.run(BaseSegmentPushJobRunner.java:149)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:150)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:118)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:130)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.pinot.tools.Command.call(Command.java:33)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.pinot.tools.Command.call(Command.java:29)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine.access$1300(CommandLine.java:145)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine.execute(CommandLine.java:2078)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
Sending request:
http://my-pinot-controller:9000/v2/segments?tableName=myTable to controller:
ip-xx-xxx-xxx-xx.my-zone.compute.internal, version: Unknown
Caught temporary exception while pushing table: myTable segment:
myTable__3__146__20230311T2306Z to http://my-pinot-controller:9000, will retry
org.apache.pinot.common.exception.HttpErrorStatusException: Got error status
code: 500 (Internal Server Error) with reason: "Exception while uploading
segment: No enum constant
org.apache.pinot.common.utils.FileUploadDownloadClient.FileUploadType.METADATA,METADATA,METADATA"
while sending request:
http://my-pinot-controller:9000/v2/segments?tableName=myTable to controller:
ip-xx-xxx-xxx-xx.my-zone.compute.internal, version: Unknown
at
org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:442)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.common.utils.FileUploadDownloadClient.uploadSegmentMetadata(FileUploadDownloadClient.java:583)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.lambda$sendSegmentUriAndMetadata$2(SegmentPushUtils.java:314)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUriAndMetadata(SegmentPushUtils.java:304)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUriAndMetadata(SegmentPushUtils.java:136)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner.uploadSegments(SegmentMetadataPushJobRunner.java:38)
[pinot-batch-ingestion-standalone-0.12.0-shaded.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.plugin.ingestion.batch.common.BaseSegmentPushJobRunner.run(BaseSegmentPushJobRunner.java:149)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:150)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:118)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:130)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.pinot.tools.Command.call(Command.java:33)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.pinot.tools.Command.call(Command.java:29)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine.access$1300(CommandLine.java:145)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine.execute(CommandLine.java:2078)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
Got exception to kick off standalone data ingestion job -
java.lang.RuntimeException: Caught exception during running -
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:152)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:118)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:130)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.pinot.tools.Command.call(Command.java:33)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at org.apache.pinot.tools.Command.call(Command.java:29)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine.access$1300(CommandLine.java:145)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at picocli.CommandLine.execute(CommandLine.java:2078)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202)
[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException:
Operation failed after 2 attempts
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUriAndMetadata(SegmentPushUtils.java:304)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUriAndMetadata(SegmentPushUtils.java:136)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner.uploadSegments(SegmentMetadataPushJobRunner.java:38)
~[pinot-batch-ingestion-standalone-0.12.0-shaded.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.plugin.ingestion.batch.common.BaseSegmentPushJobRunner.run(BaseSegmentPushJobRunner.java:149)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:150)
~[pinot-all-0.12.0-jar-with-dependencies.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
... 13 more
java.lang.RuntimeException: Caught exception during running -
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:152)
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:118)
at
org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:130)
at org.apache.pinot.tools.Command.call(Command.java:33)
at org.apache.pinot.tools.Command.call(Command.java:29)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171)
at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202)
Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException:
Operation failed after 2 attempts
at
org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65)
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUriAndMetadata(SegmentPushUtils.java:304)
at
org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUriAndMetadata(SegmentPushUtils.java:136)
at
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner.uploadSegments(SegmentMetadataPushJobRunner.java:38)
at
org.apache.pinot.plugin.ingestion.batch.common.BaseSegmentPushJobRunner.run(BaseSegmentPushJobRunner.java:149)
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:150)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
