Aryan created HADOOP-18161:
------------------------------
Summary: [WASB] Retry not getting implemented when using wasb
scheme in hadoop-azure 2.7.4
Key: HADOOP-18161
URL: https://issues.apache.org/jira/browse/HADOOP-18161
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure, hadoop-thirdparty
Affects Versions: 2.7.4
Reporter: Aryan
I am using prestodb to read data from blob.
Presto is using hadoop-azure-2.7.4 jar.
I'm using *wasb* scheme to query the data on blob. I'm afraid for some reason
the hadoop-azure library is not retrying when getting IO exception.
Attaching the stack trace below,
{code:java}
com.facebook.presto.spi.PrestoException: Error reading from
wasb://oemdpv3prd...@oemdpv3prd.blob.core.windows.net/data/pipelines/hudi/kafka/telemetrics_v2/dp.hmi.quectel.bms.data.packet.v2/dt=2022-01-15/e576abc3-942a-434d-be02-6899798258eb-0_5-13327-290407_20220115211203.parquet
at position 65924529
at
com.facebook.presto.hive.parquet.HdfsParquetDataSource.readInternal(HdfsParquetDataSource.java:66)
at
com.facebook.presto.parquet.AbstractParquetDataSource.readFully(AbstractParquetDataSource.java:60)
at
com.facebook.presto.parquet.AbstractParquetDataSource.readFully(AbstractParquetDataSource.java:51)
at
com.facebook.presto.parquet.reader.ParquetReader.readPrimitive(ParquetReader.java:247)
at
com.facebook.presto.parquet.reader.ParquetReader.readColumnChunk(ParquetReader.java:330)
at
com.facebook.presto.parquet.reader.ParquetReader.readBlock(ParquetReader.java:313)
at
com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:182)
at
com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:160)
at
com.facebook.presto.common.block.LazyBlock.assureLoaded(LazyBlock.java:291)
at
com.facebook.presto.common.block.LazyBlock.getLoadedBlock(LazyBlock.java:282)
at
com.facebook.presto.operator.ScanFilterAndProjectOperator$RecordingLazyBlockLoader.load(ScanFilterAndProjectOperator.java:314)
at
com.facebook.presto.operator.ScanFilterAndProjectOperator$RecordingLazyBlockLoader.load(ScanFilterAndProjectOperator.java:300)
at
com.facebook.presto.common.block.LazyBlock.assureLoaded(LazyBlock.java:291)
at
com.facebook.presto.common.block.LazyBlock.getLoadedBlock(LazyBlock.java:282)
at
com.facebook.presto.operator.project.InputPageProjection.project(InputPageProjection.java:69)
at
com.facebook.presto.operator.project.PageProjectionWithOutputs.project(PageProjectionWithOutputs.java:56)
at
com.facebook.presto.operator.project.PageProcessor$ProjectSelectedPositions.processBatch(PageProcessor.java:323)
at
com.facebook.presto.operator.project.PageProcessor$ProjectSelectedPositions.process(PageProcessor.java:197)
at
com.facebook.presto.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:315)
at
com.facebook.presto.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:79)
at
com.facebook.presto.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:65)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at
com.facebook.presto.operator.project.MergingPageOutput.getOutput(MergingPageOutput.java:113)
at
com.facebook.presto.operator.ScanFilterAndProjectOperator.processPageSource(ScanFilterAndProjectOperator.java:295)
at
com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:242)
at com.facebook.presto.operator.Driver.processInternal(Driver.java:418)
at com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:301)
at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:722)
at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
at
com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
at
com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
at
com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:599)
at
com.facebook.presto.$gen.Presto_0_256_2ed7f73____20220112_133056_1.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: undefined
at
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:598)
at
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:264)
at
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448)
at
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:735)
at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:65)
at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:78)
at
org.apache.hadoop.fs.BufferedFSInputStream.readFully(BufferedFSInputStream.java:113)
at
org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:107)
at
com.facebook.presto.hive.parquet.HdfsParquetDataSource.readInternal(HdfsParquetDataSource.java:58)
... 36 more
Caused by:
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.StorageException:
Blob hash mismatch (integrity check failed), Expected value is
gRfj04hQvdOqOaAdqo+lGw==, retrieved 4V4j3H2KabAsssDyGnJN9w==.
at
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.CloudBlob$10.postProcessResponse(CloudBlob.java:1714)
at
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.CloudBlob$10.postProcessResponse(CloudBlob.java:1613)
at
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:148)
at
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1468)
at
com.facebook.presto.hadoop.$internal.com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255)
... 47 more{code}
Please suggest a way to get the retry implemented.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
