* If the file exists but is a directory rather than a regular file, does * not exist but cannot be created, or cannot be opened for any other * reason then a <code>FileNotFoundException</code> is thrown.
After searching into FileOutputStream i saw this annotation.So you can check executor node first(may be no permission or no space,or no enough file descriptor) 2017-07-25 13:05 GMT+08:00 周康 <zhoukang199...@gmail.com>: > You can also check whether space left in the executor node enough to store > shuffle file or not. > > 2017-07-25 13:01 GMT+08:00 周康 <zhoukang199...@gmail.com>: > >> First,spark will handle task fail so if job ended normally , this error >> can be ignore. >> Second, when using BypassMergeSortShuffleWriter, it will first write data >> file then write an index file. >> You can check "Failed to delete temporary index file at" or "fail to >> rename file" in related executor node's log file. >> >> 2017-07-25 0:33 GMT+08:00 Martin Peng <wei...@gmail.com>: >> >>> Is there anyone at share me some lights about this issue? >>> >>> Thanks >>> Martin >>> >>> 2017-07-21 18:58 GMT-07:00 Martin Peng <wei...@gmail.com>: >>> >>>> Hi, >>>> >>>> I have several Spark jobs including both batch job and Stream jobs to >>>> process the system log and analyze them. We are using Kafka as the pipeline >>>> to connect each jobs. >>>> >>>> Once upgrade to Spark 2.1.0 + Spark Kafka Streaming 010, I found some >>>> of the jobs(both batch or streaming) are thrown below exceptions >>>> randomly(either after several hours run or just run in 20 mins). Can anyone >>>> give me some suggestions about how to figure out the real root cause? >>>> (Looks like google result is not very useful...) >>>> >>>> Thanks, >>>> Martin >>>> >>>> 00:30:04,510 WARN - 17/07/22 00:30:04 WARN TaskSetManager: Lost task >>>> 60.0 in stage 1518490.0 (TID 338070, 10.133.96.21, executor 0): >>>> java.io.FileNotFoundException: /mnt/mesos/work_dir/slaves/201 >>>> 60924-021501-274760970-5050-7646-S2/frameworks/40aeb8e5-e82a >>>> -4df9-b034-8815a7a7564b-2543/executors/0/runs/fd15c15d-2511- >>>> 4f37-a106-27431f583153/blockmgr-a0e0e673-f88b-4d12-a802- >>>> c35643e6c6b2/33/shuffle_2090_60_0.index.b66235be-79be-4455-9759-1c7ba70f91f6 >>>> (No such file or directory) >>>> 00:30:04,510 WARN - at java.io.FileOutputStream.open0(Native >>>> Method) >>>> 00:30:04,510 WARN - at java.io.FileOutputStream.open( >>>> FileOutputStream.java:270) >>>> 00:30:04,510 WARN - at java.io.FileOutputStream.<init >>>> >(FileOutputStream.java:213) >>>> 00:30:04,510 WARN - at java.io.FileOutputStream.<init >>>> >(FileOutputStream.java:162) >>>> 00:30:04,510 WARN - at org.apache.spark.shuffle.Index >>>> ShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlo >>>> ckResolver.scala:144) >>>> 00:30:04,510 WARN - at org.apache.spark.shuffle.sort. >>>> BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWri >>>> ter.java:128) >>>> 00:30:04,510 WARN - at org.apache.spark.scheduler.Shu >>>> ffleMapTask.runTask(ShuffleMapTask.scala:96) >>>> 00:30:04,510 WARN - at org.apache.spark.scheduler.Shu >>>> ffleMapTask.runTask(ShuffleMapTask.scala:53) >>>> 00:30:04,510 WARN - at org.apache.spark.scheduler.Tas >>>> k.run(Task.scala:99) >>>> 00:30:04,510 WARN - at org.apache.spark.executor.Exec >>>> utor$TaskRunner.run(Executor.scala:282) >>>> 00:30:04,510 WARN - at java.util.concurrent.ThreadPoo >>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> 00:30:04,510 WARN - at java.util.concurrent.ThreadPoo >>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> 00:30:04,510 WARN - at java.lang.Thread.run(Thread.java:748) >>>> >>>> 00:30:04,580 INFO - Driver stacktrace: >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAGScheduler.org >>>> $apache$spark$scheduler$DAGScheduler$$failJobAn >>>> dIndependentStages(DAGScheduler.scala:1435) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> Scheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> Scheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422) >>>> 00:30:04,580 INFO - scala.collection.mutable.Resiz >>>> ableArray$class.foreach(ResizableArray.scala:59) >>>> 00:30:04,580 INFO - scala.collection.mutable.Array >>>> Buffer.foreach(ArrayBuffer.scala:48) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> Scheduler.abortStage(DAGScheduler.scala:1422) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> Scheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> Scheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) >>>> 00:30:04,580 INFO - scala.Option.foreach(Option.scala:257) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> Scheduler.handleTaskSetFailed(DAGScheduler.scala:802) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> SchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> SchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> SchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594) >>>> 00:30:04,580 INFO - org.apache.spark.util.EventLoo >>>> p$$anon$1.run(EventLoop.scala:48) >>>> 00:30:04,580 INFO - org.apache.spark.scheduler.DAG >>>> Scheduler.runJob(DAGScheduler.scala:628) >>>> 00:30:04,580 INFO - org.apache.spark.SparkContext. >>>> runJob(SparkContext.scala:1918) >>>> 00:30:04,580 INFO - org.apache.spark.SparkContext. >>>> runJob(SparkContext.scala:1931) >>>> 00:30:04,580 INFO - org.apache.spark.SparkContext. >>>> runJob(SparkContext.scala:1944) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDD$$anon >>>> fun$take$1.apply(RDD.scala:1353) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDDOperat >>>> ionScope$.withScope(RDDOperationScope.scala:151) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDDOperat >>>> ionScope$.withScope(RDDOperationScope.scala:112) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDD.withScope(RDD.scala:362) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDD.take(RDD.scala:1326) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDD$$anon >>>> fun$isEmpty$1.apply$mcZ$sp(RDD.scala:1461) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDD$$anon >>>> fun$isEmpty$1.apply(RDD.scala:1461) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDD$$anon >>>> fun$isEmpty$1.apply(RDD.scala:1461) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDDOperat >>>> ionScope$.withScope(RDDOperationScope.scala:151) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDDOperat >>>> ionScope$.withScope(RDDOperationScope.scala:112) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDD.withScope(RDD.scala:362) >>>> 00:30:04,580 INFO - org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1460) >>>> 00:30:04,580 INFO - com.ericsson.mediafirst.spark. >>>> clientlogsenrichment.ClientLogsEnrichmentJob$.executeIterati >>>> on(ClientLogsEnrichmentJob.scala:133) >>>> 00:30:04,580 INFO - com.ericsson.mediafirst.spark. >>>> clientlogsenrichment.ClientLogsEnrichmentJob$.runIteration(C >>>> lientLogsEnrichmentJob.scala:76) >>>> 00:30:04,581 INFO - com.ericsson.mediafirst.spark. >>>> clientlogsenrichment.ClientLogsEnrichmentJob$.runBatch(Clien >>>> tLogsEnrichmentJob.scala:59) >>>> 00:30:04,581 INFO - com.ericsson.mediafirst.sparku >>>> tils.jobtemplates.BatchJob.main(BatchJob.scala:35) >>>> 00:30:04,581 INFO - com.ericsson.mediafirst.spark. >>>> clientlogsenrichment.ClientLogsEnrichmentJob.main(ClientLogs >>>> EnrichmentJob.scala) >>>> 00:30:04,581 INFO - sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>> Method) >>>> 00:30:04,581 INFO - sun.reflect.NativeMethodAccess >>>> orImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> 00:30:04,581 INFO - sun.reflect.DelegatingMethodAc >>>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> 00:30:04,581 INFO - java.lang.reflect.Method.invoke(Method.java:498) >>>> 00:30:04,581 INFO - org.apache.spark.deploy.SparkS >>>> ubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSub >>>> mit.scala:738) >>>> 00:30:04,581 INFO - org.apache.spark.deploy.SparkS >>>> ubmit$.doRunMain$1(SparkSubmit.scala:187) >>>> 00:30:04,581 INFO - org.apache.spark.deploy.SparkS >>>> ubmit$.submit(SparkSubmit.scala:212) >>>> 00:30:04,581 INFO - org.apache.spark.deploy.SparkS >>>> ubmit$.main(SparkSubmit.scala:126) >>>> 00:30:04,581 INFO - org.apache.spark.deploy.SparkS >>>> ubmit.main(SparkSubmit.scala) >>>> 00:30:04,581 WARN - 17/07/22 00:30:04 WARN JobProgressListener: Task >>>> start for unknown stage 1518491 >>>> 00:30:04,670 WARN - 17/07/22 00:30:04 ERROR LiveListenerBus: >>>> SparkListenerBus has already stopped! Dropping event >>>> SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(0, >>>> 10.133.96.21, 45377, None),rdd_15721_0,StorageLevel(memory, >>>> deserialized, 1 replicas),12024,0)) >>>> 00:30:04,673 WARN - 17/07/22 00:30:04 ERROR LiveListenerBus: >>>> SparkListenerBus has already stopped! Dropping event >>>> SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(0, >>>> 10.133.96.21, 45377, None),rdd_15721_1,StorageLevel(memory, >>>> deserialized, 1 replicas),13736,0)) >>>> 00:30:04,679 WARN - 17/07/22 00:30:04 ERROR TransportRequestHandler: >>>> Error while invoking RpcHandler#receive() for one-way message. >>>> 00:30:04,679 WARN - org.apache.spark.SparkException: Could not find >>>> CoarseGrainedScheduler. >>>> 00:30:04,679 WARN - at org.apache.spark.rpc.netty.Dis >>>> patcher.postMessage(Dispatcher.scala:154) >>>> 00:30:04,679 WARN - at org.apache.spark.rpc.netty.Dis >>>> patcher.postOneWayMessage(Dispatcher.scala:134) >>>> 00:30:04,679 WARN - at org.apache.spark.rpc.netty.Net >>>> tyRpcHandler.receive(NettyRpcEnv.scala:570) >>>> 00:30:04,679 WARN - at org.apache.spark.network.serve >>>> r.TransportRequestHandler.processOneWayMessage(TransportRequ >>>> estHandler.java:180) >>>> 00:30:04,679 WARN - at org.apache.spark.network.serve >>>> r.TransportRequestHandler.handle(TransportRequestHandler.java:109) >>>> 00:30:04,679 WARN - at org.apache.spark.network.serve >>>> r.TransportChannelHandler.channelRead0(TransportChannelHandl >>>> er.java:119) >>>> 00:30:04,679 WARN - at org.apache.spark.network.serve >>>> r.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) >>>> 00:30:04,679 WARN - at io.netty.channel.SimpleChannel >>>> InboundHandler.channelRead(SimpleChannelInboundHandler.java:105) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.fireChannelRead(AbstractChannelHandlerConte >>>> xt.java:346) >>>> 00:30:04,679 WARN - at io.netty.handler.timeout.IdleS >>>> tateHandler.channelRead(IdleStateHandler.java:266) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.fireChannelRead(AbstractChannelHandlerConte >>>> xt.java:346) >>>> 00:30:04,679 WARN - at io.netty.handler.codec.Message >>>> ToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.fireChannelRead(AbstractChannelHandlerConte >>>> xt.java:346) >>>> 00:30:04,679 WARN - at org.apache.spark.network.util. >>>> TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.fireChannelRead(AbstractChannelHandlerConte >>>> xt.java:346) >>>> 00:30:04,679 WARN - at io.netty.channel.DefaultChanne >>>> lPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.DefaultChanne >>>> lPipeline.fireChannelRead(DefaultChannelPipeline.java:911) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.AbstractN >>>> ioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.NioEventL >>>> oop.processSelectedKey(NioEventLoop.java:652) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.NioEventL >>>> oop.processSelectedKeysOptimized(NioEventLoop.java:575) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.NioEventL >>>> oop.processSelectedKeys(NioEventLoop.java:489) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.NioEventL >>>> oop.run(NioEventLoop.java:451) >>>> 00:30:04,679 WARN - at io.netty.util.concurrent.Singl >>>> eThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) >>>> 00:30:04,679 WARN - at io.netty.util.concurrent.Defau >>>> ltThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFa >>>> ctory.java:144) >>>> 00:30:04,679 WARN - at java.lang.Thread.run(Thread.java:748) >>>> 00:30:04,679 WARN - 17/07/22 00:30:04 ERROR TransportRequestHandler: >>>> Error while invoking RpcHandler#receive() for one-way message. >>>> 00:30:04,679 WARN - org.apache.spark.SparkException: Could not find >>>> CoarseGrainedScheduler. >>>> 00:30:04,679 WARN - at org.apache.spark.rpc.netty.Dis >>>> patcher.postMessage(Dispatcher.scala:154) >>>> 00:30:04,679 WARN - at org.apache.spark.rpc.netty.Dis >>>> patcher.postOneWayMessage(Dispatcher.scala:134) >>>> 00:30:04,679 WARN - at org.apache.spark.rpc.netty.Net >>>> tyRpcHandler.receive(NettyRpcEnv.scala:570) >>>> 00:30:04,679 WARN - at org.apache.spark.network.serve >>>> r.TransportRequestHandler.processOneWayMessage(TransportRequ >>>> estHandler.java:180) >>>> 00:30:04,679 WARN - at org.apache.spark.network.serve >>>> r.TransportRequestHandler.handle(TransportRequestHandler.java:109) >>>> 00:30:04,679 WARN - at org.apache.spark.network.serve >>>> r.TransportChannelHandler.channelRead0(TransportChannelHandl >>>> er.java:119) >>>> 00:30:04,679 WARN - at org.apache.spark.network.serve >>>> r.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) >>>> 00:30:04,679 WARN - at io.netty.channel.SimpleChannel >>>> InboundHandler.channelRead(SimpleChannelInboundHandler.java:105) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.fireChannelRead(AbstractChannelHandlerConte >>>> xt.java:346) >>>> 00:30:04,679 WARN - at io.netty.handler.timeout.IdleS >>>> tateHandler.channelRead(IdleStateHandler.java:266) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.fireChannelRead(AbstractChannelHandlerConte >>>> xt.java:346) >>>> 00:30:04,679 WARN - at io.netty.handler.codec.Message >>>> ToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.fireChannelRead(AbstractChannelHandlerConte >>>> xt.java:346) >>>> 00:30:04,679 WARN - at org.apache.spark.network.util. >>>> TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.fireChannelRead(AbstractChannelHandlerConte >>>> xt.java:346) >>>> 00:30:04,679 WARN - at io.netty.channel.DefaultChanne >>>> lPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:367) >>>> 00:30:04,679 WARN - at io.netty.channel.AbstractChann >>>> elHandlerContext.invokeChannelRead(AbstractChannelHandlerCon >>>> text.java:353) >>>> 00:30:04,679 WARN - at io.netty.channel.DefaultChanne >>>> lPipeline.fireChannelRead(DefaultChannelPipeline.java:911) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.AbstractN >>>> ioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.NioEventL >>>> oop.processSelectedKey(NioEventLoop.java:652) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.NioEventL >>>> oop.processSelectedKeysOptimized(NioEventLoop.java:575) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.NioEventL >>>> oop.processSelectedKeys(NioEventLoop.java:489) >>>> 00:30:04,679 WARN - at io.netty.channel.nio.NioEventL >>>> oop.run(NioEventLoop.java:451) >>>> 00:30:04,679 WARN - at io.netty.util.concurrent.Singl >>>> eThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) >>>> 00:30:04,679 WARN - at io.netty.util.concurrent.Defau >>>> ltThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFa >>>> ctory.java:144) >>>> 00:30:04,679 WARN - at java.lang.Thread.run(Thread.java:748) >>>> 00:30:11,318 WARN - I0722 00:30:11.318724 2921 sched.cpp:2021] Asked >>>> to stop the driver >>>> 00:30:11,318 WARN - I0722 00:30:11.318838 2988 sched.cpp:1203] >>>> Stopping framework 40aeb8e5-e82a-4df9-b034-8815a7a7564b-2543 >>>> >>> >>> >> >
