[jira] [Updated] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7475: -- Fix Version/s: 3.5.0 > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7475: -- Affects Version/s: 3.4.0 > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned MAPREDUCE-7475: - Assignee: Kaiyao Ke > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7475: -- Priority: Minor (was: Major) > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Minor > Labels: pull-request-available > Fix For: 3.5.0 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7475) 2 tests are non-idempotent (passes in the first run but fails in repeated runs in the same JVM)
[ https://issues.apache.org/jira/browse/MAPREDUCE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7475: -- Component/s: test > 2 tests are non-idempotent (passes in the first run but fails in repeated > runs in the same JVM) > --- > > Key: MAPREDUCE-7475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 > Environment: Ubuntu 22.04, Java 17 >Reporter: Kaiyao Ke >Assignee: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > Original Estimate: 10m > Remaining Estimate: 10m > > 2 tests are not idempotent and fails upon repeated execution within the same > JVM instance due to self-induced state pollution. Specifically, these tests > try to make the directory TEST_ROOT_DIR and write to it. The tests do not > clean up (remove) the directory after execution. Therefore, in the second > execution, TEST_ROOT_DIR would already exist and the exception `Could not > create test dir` would be thrown. Below are the 2 non-idempotent tests: > * org.apache.hadoop.mapred.TestOldCombinerGrouping.testCombiner > * org.apache.hadoop.mapreduce.TestNewCombinerGrouping.testCombiner -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7474. --- Fix Version/s: 3.3.9 3.5.0 3.4.1 Resolution: Fixed > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9, 3.5.0, 3.4.1 > > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed? > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7476) Follow up of MAPREDUCE-7475 - fix more non-idempotent tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-7476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7476: -- Summary: Follow up of MAPREDUCE-7475 - fix more non-idempotent tests (was: Follow up of https://issues.apache.org/jira/browse/MAPREDUCE-7475 - detected 5 more non-idempotent tests(pass in the first run but fails in repeated runs in the same JVM)) > Follow up of MAPREDUCE-7475 - fix more non-idempotent tests > --- > > Key: MAPREDUCE-7476 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7476 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Kaiyao Ke >Priority: Major > Labels: pull-request-available > Original Estimate: 1h > Remaining Estimate: 1h > > Similar with https://issues.apache.org/jira/browse/MAPREDUCE-7475 , 5 more > non-idempotent unit tests are detected. > The following two tests below do not reset `NotificationServlet.counter`, so > repeated runs throw assertion failures due to accumulation: > * org.apache.hadoop.mapred.TestClusterMRNotification#testMR > * org.apache.hadoop.mapred.TestLocalMRNotification#testMR > The following test does not remove the key `AMParams.ATTEMPT_STATE`, so > repeated runs of the test will not be missing the attempt-state at all: > * org.apache.hadoop.mapreduce.v2.app.webapp.TestAppController.testAttempts > The following test fully deletes `TEST_ROOT_DIR` after execution, so repeated > runs will throw a`DiskErrorException`: > * org.apache.hadoop.mapred.TestMapTask#testShufflePermissions > The following test does not restore the static variable `statusUpdateTimes` > after execution, so consecutive runs throws `AssertionError`: > * org.apache.hadoop.mapred.TestTaskProgressReporter#testTaskProgress -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7474: -- Description: * Manifest committer is not resilient to rename failures on task commit without HADOOP-18012 rename recovery enabled. * large burst of delete calls noted: are they needed? relates to HADOOP-19093 but takes a more minimal approach with goal of changes in manifest committer only. Initial proposed changes * retry recovery on task commit rename, always (repeat save, delete, rename) * audit delete use and see if it can be pruned was: * Manifest committer is not resilient to rename failures on task commit without HADOOP-18012 rename recovery enabled. * large burst of delete calls noted: are they needed relates to HADOOP-19093 but takes a more minimal approach with goal of changes in manifest committer only. Initial proposed changes * retry recovery on task commit rename, always (repeat save, delete, rename) * audit delete use and see if it can be pruned * maybe: rate limit some IO internally, but not delegate to abfs > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed? > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
Steve Loughran created MAPREDUCE-7474: - Summary: [ABFS] Improve commit resilience and performance in Manifest Committer Key: MAPREDUCE-7474 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.3.6, 3.4.0 Reporter: Steve Loughran * Manifest committer is not resilient to rename failures on task commit without HADOOP-18012 rename recovery enabled. * large burst of delete calls noted: are they needed relates to HADOOP-19093 but takes a more minimal approach with goal of changes in manifest committer only. Initial proposed changes * retry recovery on task commit rename, always (repeat save, delete, rename) * audit delete use and see if it can be pruned * maybe: rate limit some IO internally, but not delegate to abfs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7474) [ABFS] Improve commit resilience and performance in Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned MAPREDUCE-7474: - Assignee: Steve Loughran > [ABFS] Improve commit resilience and performance in Manifest Committer > -- > > Key: MAPREDUCE-7474 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 3.4.0, 3.3.6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > * Manifest committer is not resilient to rename failures on task commit > without HADOOP-18012 rename recovery enabled. > * large burst of delete calls noted: are they needed > relates to HADOOP-19093 but takes a more minimal approach with goal of > changes in manifest committer only. > Initial proposed changes > * retry recovery on task commit rename, always (repeat save, delete, rename) > * audit delete use and see if it can be pruned > * maybe: rate limit some IO internally, but not delegate to abfs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7470) multi-thread mapreduce committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7470. --- Resolution: Duplicate > multi-thread mapreduce committer > > > Key: MAPREDUCE-7470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7470 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: TianyiMa >Priority: Major > Labels: mapreduce, pull-request-available > Attachments: MAPREDUCE-7470.0.patch > > > In cloud environment, such as aws, aliyun etc., the internet delay is > non-trival when we commit thounds of files. > In our situation, the ping delay is about 0.03ms in IDC, but when move to > Coud, the ping delay is about 3ms, which is roughly 100x slower. We found > that, committing tens thounds of files will cost a few tens of minutes. The > more files there are, the logger it takes. > So we propose a new committer algorithm, which is a variant of committer > algorithm version 1, called 3. In this new algorithm 3, in order to decrease > the committer time, we use a thread pool to commit job's final output. > Our test result in Cloud production shows that, the new algorithm 3 has > decrease the committer time by serveral tens of times. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7470) multi-thread mapreduce v1 FileOutputcommitter
[ https://issues.apache.org/jira/browse/MAPREDUCE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7470: -- Summary: multi-thread mapreduce v1 FileOutputcommitter (was: multi-thread mapreduce committer) > multi-thread mapreduce v1 FileOutputcommitter > - > > Key: MAPREDUCE-7470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7470 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: TianyiMa >Priority: Major > Labels: mapreduce, pull-request-available > Attachments: MAPREDUCE-7470.0.patch > > > In cloud environment, such as aws, aliyun etc., the internet delay is > non-trival when we commit thounds of files. > In our situation, the ping delay is about 0.03ms in IDC, but when move to > Coud, the ping delay is about 3ms, which is roughly 100x slower. We found > that, committing tens thounds of files will cost a few tens of minutes. The > more files there are, the logger it takes. > So we propose a new committer algorithm, which is a variant of committer > algorithm version 1, called 3. In this new algorithm 3, in order to decrease > the committer time, we use a thread pool to commit job's final output. > Our test result in Cloud production shows that, the new algorithm 3 has > decrease the committer time by serveral tens of times. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7465) performance problem in FileOutputCommitter for big list processed by single thread
[ https://issues.apache.org/jira/browse/MAPREDUCE-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7465: -- Summary: performance problem in FileOutputCommitter for big list processed by single thread (was: performance problem in FileOutputCommiter for big list processed by single thread) > performance problem in FileOutputCommitter for big list processed by single > thread > --- > > Key: MAPREDUCE-7465 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7465 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance >Affects Versions: 3.2.3, 3.3.2, 3.2.4, 3.3.5, 3.3.3, 3.3.4, 3.3.6 >Reporter: Arnaud Nauwynck >Priority: Minor > Labels: pull-request-available > > when commiting a big hadoop job (for example via Spark) having many > partitions, > the class FileOutputCommiter process thousands of dirs/files to rename with a > single Thread. This is performance issue, caused by lot of waits on > FileStystem storage operations. > I propose that above a configurable threshold (default=3, configurable via > property 'mapreduce.fileoutputcommitter.parallel.threshold'), the class > FileOutputCommiter process the list of files to rename using parallel > threads, using the default jvm ExecutorService (ForkJoinPool.commonPool()) > See Pull-Request: > [https://github.com/apache/hadoop/pull/6378|https://github.com/apache/hadoop/pull/6378] > Notice that sub-class instances of FileOutputCommiter are supposed to be > created at runtime dependending of a configurable property > ([https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PathOutputCommitterFactory.java|PathOutputCommitterFactory.java]). > But for example in Parquet + Spark, this is buggy and can not be changed at > runtime. > There is an ongoing Jira and PR to fix it in Parquet + Spark: > [https://issues.apache.org/jira/browse/PARQUET-2416|https://issues.apache.org/jira/browse/PARQUET-2416] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7471) Hadoop mapred minicluster command line fails with class not found
[
https://issues.apache.org/jira/browse/MAPREDUCE-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17813156#comment-17813156
]
Steve Loughran commented on MAPREDUCE-7471:
---
* for at getBaseDirectory() I think we should move off junit for that specific
assertion; straightforward enough.
* that mockito thing still happens on trunk.
the hadoop-client-minicluster module aims to hide all the transitive
dependencies for downstream testing; looks like the bin/mr/minicluster command
still needs it
I don't want mockito on the classpath if it is at all externally visible.
> Hadoop mapred minicluster command line fails with class not found
> -
>
> Key: MAPREDUCE-7471
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7471
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.3.5
>Reporter: Duo Zhang
>Priority: Major
>
> If you run
> ./bin/mapred minicluster
> It will fail with
> {noformat}
> Exception in thread "Listener at localhost/35325"
> java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2648)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2662)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1510)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:989)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:588)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:530)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:160)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
> Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 9 more
> {noformat}
> This line
> https://github.com/apache/hadoop/blob/835403d872506c4fa76eb2d721f2d91f413473d5/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java#L2648
> This is because we rely on mockito in NameNodeAdapter but we do not have
> mockito on our classpath, at least in our published hadoop-3.3.5 binary.
> And there is another problem that, if we do not run the above command in the
> HADOOP_HOME directory, i.e, in another directory by typing the absolute path
> of the mapred command, it will fail with
> {noformat}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:336)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:280)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:289)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:3069)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.(MiniDFSCluster.java:239)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:157)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
> at
> org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 8 mor
> {noformat}
> This simply because this line
> https://github.com/apache/hadoop/blob/835403d872506c4fa76eb2d721f2d91f413473d5/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L601
> We should add the $HADOOP_TOOLS_HOME prefix for the default value of
> HADOOP_TOOLS_DIR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7470) multi-thread mapreduce committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811557#comment-17811557 ] Steve Loughran commented on MAPREDUCE-7470: --- This is a duplicate of MAPREDUCE-7465, and I don't want it in for the same reasons: scale and correctness. Read: https://github.com/steveloughran/zero-rename-committer for background h3. V1: * relies on atomic directory rename for task commit. * Slow and broken task commit on s3 O(data) * O(files) and broken task commit on google * awfully slow job commit (s3, again O(data)) h3. v2 broken everywhere unable to cope with failure of task commit during nonatomic commit process. S3A ships with a zero rename committer which uploads directly to the destination but doesn't complete the rename until job commit (O(files/threads) commit We didn't implement this directly in FileOutputCommitter because of fear of braking things. That's a critical piece of code and it has two different intermingled algorithms and about the only place I've ever come across co-recursion in production. It's complex and brittle. Instead we added the ability to define a committer factory for different filesystems. h3. Manifest Committer MAPREDUCE-7341 added an "intermediate manifest committer", based on * the s3a committer experience * benchmarking abfs including how it fails under load * need to make google jobs safe Task commit: treewalk task attempt dir, save a manifest of files to rename and dirs to creste. Moves a lot of the IO to task commit, which really matters for abfs. safe on GCS. Job commit: parallel load of manifests, dir creation phase and then parallel rename * fast parallel load of manifests * parallel rename O(files/threads) * uses etag tracking for recovery from throttle-initiated rename failures on abfs (which is why multithreaded job commit doesn't work reliably there: they can do so much IO that throttling kicks off. Also lets you rate limit * collects and saves statistics to the JSON file; same format as the s3a committers This is in hadoop 3.3.5+. Yes, it's more complex than parallel threads, but we wrote it *because we tried the parallel design and it didn't work*. FWIW, here's our PR for parallel runs: https://github.com/apache/hadoop/pull/6399 Arnaud's is pretty similar: https://github.com/apache/hadoop/pull/6378 both of these (like yours) will overload azure storage on big jobs/multiple jobs in same storage account committing; none of these prs are safe for GCS. I don't know about allyun's correctness here. Can you try the manifest committer, see how it goes and report any issues. Sharing the _SUCCESS file would be nice too > multi-thread mapreduce committer > > > Key: MAPREDUCE-7470 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7470 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: TianyiMa >Priority: Major > Labels: mapreduce, pull-request-available > Attachments: MAPREDUCE-7470.0.patch > > > In cloud environment, such as aws, aliyun etc., the internet delay is > non-trival when we commit thounds of files. > In our situation, the ping delay is about 0.03ms in IDC, but when move to > Coud, the ping delay is about 3ms, which is roughly 100x slower. We found > that, committing tens thounds of files will cost a few tens of minutes. The > more files there are, the logger it takes. > So we propose a new committer algorithm, which is a variant of committer > algorithm version 1, called 3. In this new algorithm 3, in order to decrease > the committer time, we use a thread pool to commit job's final output. > Our test result in Cloud production shows that, the new algorithm 3 has > decrease the committer time by serveral tens of times. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7465) performance problem in FileOutputCommiter for big list processed by single thread
[ https://issues.apache.org/jira/browse/MAPREDUCE-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17802259#comment-17802259 ] Steve Loughran commented on MAPREDUCE-7465: --- [~arnaud.nauwynck]: https://spark.apache.org/docs/latest/cloud-integration.html#committing-work-into-cloud-storage-safely-and-fast > performance problem in FileOutputCommiter for big list processed by single > thread > -- > > Key: MAPREDUCE-7465 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7465 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance >Affects Versions: 3.2.3, 3.3.2, 3.2.4, 3.3.5, 3.3.3, 3.3.4, 3.3.6 >Reporter: Arnaud Nauwynck >Priority: Minor > Labels: pull-request-available > > when commiting a big hadoop job (for example via Spark) having many > partitions, > the class FileOutputCommiter process thousands of dirs/files to rename with a > single Thread. This is performance issue, caused by lot of waits on > FileStystem storage operations. > I propose that above a configurable threshold (default=3, configurable via > property 'mapreduce.fileoutputcommitter.parallel.threshold'), the class > FileOutputCommiter process the list of files to rename using parallel > threads, using the default jvm ExecutorService (ForkJoinPool.commonPool()) > See Pull-Request: > [https://github.com/apache/hadoop/pull/6378|https://github.com/apache/hadoop/pull/6378] > Notice that sub-class instances of FileOutputCommiter are supposed to be > created at runtime dependending of a configurable property > ([https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PathOutputCommitterFactory.java|PathOutputCommitterFactory.java]). > But for example in Parquet + Spark, this is buggy and can not be changed at > runtime. > There is an ongoing Jira and PR to fix it in Parquet + Spark: > [https://issues.apache.org/jira/browse/PARQUET-2416|https://issues.apache.org/jira/browse/PARQUET-2416] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7465) performance problem in FileOutputCommiter for big list processed by single thread
[ https://issues.apache.org/jira/browse/MAPREDUCE-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17802213#comment-17802213 ] Steve Loughran commented on MAPREDUCE-7465: --- bq. As of now, I have seen that the class "ParquetOutputformat" is hardcoded in spark, and is hard-coding the use of (extends) class "FileOutputCommiter". I did not see any way to make it configurable. It requires more PR in parquet library. spark's hadoop cloud library has the workaround for this; subclass of ParquetOutputCommitter which dynamically binds to the underlying target config. It's exactly the same as needed for the s3a committers, so any docs there will help. email me at stevel@apache and I'll help you get set up. > performance problem in FileOutputCommiter for big list processed by single > thread > -- > > Key: MAPREDUCE-7465 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7465 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance >Affects Versions: 3.2.3, 3.3.2, 3.2.4, 3.3.5, 3.3.3, 3.3.4, 3.3.6 >Reporter: Arnaud Nauwynck >Priority: Minor > Labels: pull-request-available > > when commiting a big hadoop job (for example via Spark) having many > partitions, > the class FileOutputCommiter process thousands of dirs/files to rename with a > single Thread. This is performance issue, caused by lot of waits on > FileStystem storage operations. > I propose that above a configurable threshold (default=3, configurable via > property 'mapreduce.fileoutputcommitter.parallel.threshold'), the class > FileOutputCommiter process the list of files to rename using parallel > threads, using the default jvm ExecutorService (ForkJoinPool.commonPool()) > See Pull-Request: > [https://github.com/apache/hadoop/pull/6378|https://github.com/apache/hadoop/pull/6378] > Notice that sub-class instances of FileOutputCommiter are supposed to be > created at runtime dependending of a configurable property > ([https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PathOutputCommitterFactory.java|PathOutputCommitterFactory.java]). > But for example in Parquet + Spark, this is buggy and can not be changed at > runtime. > There is an ongoing Jira and PR to fix it in Parquet + Spark: > [https://issues.apache.org/jira/browse/PARQUET-2416|https://issues.apache.org/jira/browse/PARQUET-2416] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7465) performance problem in FileOutputCommiter for big list processed by single thread
[ https://issues.apache.org/jira/browse/MAPREDUCE-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801918#comment-17801918 ] Steve Loughran commented on MAPREDUCE-7465: --- bq. I understand that developpers are reluctant to integrate the PR, but it does solve my problem correctly. I would appreciate that it is disabled by default, but configurable with a property, so I can enable it and still use the official version without local patch applied to my jars. i really, really really don't want to do this as (a) its a critical piece of code and (b) we'd be committing to maintaining it. bq. There is no problem of "controlling the throttling" in Hadoop Azure Abfss code... it is already very bad at using legacy class java.net.HttpURLConnection, so establishing very slow https connection 1 by 1 without keeping TCP sockets alive. bq. We do have throthling however, not from a single JVM, but because we have so many (>= 1000) spark applications running concurrently, and so many useless "Prefetching Threads" ! Trying to control throtling on a single JVM is in my opinion useless. Azure Abfss can support 20 Millions operations per hours (and per StorageAccount), and Microsoft Azure was even able to increase it more. bq. trying to control throtling on a single JVM is in my opinion useless. we see scale issues in job commits from renames...if you aren't seeing them then it's because if our attempts to handle this (HADOOP-18002 and related). Abfs also supports 100-continue, self-throttling and other things, for cross-JVM throttling. better than the s3a code by a large margin, where even large directory deletions can blow up and make a mess of retries within the aws sdk itself. it's precisely because of rename scale issues during job commit that the manifest committer was written and is used in production ABFS deployments. If you are targeting abfs storage, it's the way to get robust job commit even on heavily loaded stores. And it is shipping in current releases. If you are having problems getting it working, happy to assist you getting set up. > performance problem in FileOutputCommiter for big list processed by single > thread > -- > > Key: MAPREDUCE-7465 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7465 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance >Affects Versions: 3.2.3, 3.3.2, 3.2.4, 3.3.5, 3.3.3, 3.3.4, 3.3.6 >Reporter: Arnaud Nauwynck >Priority: Minor > Labels: pull-request-available > > when commiting a big hadoop job (for example via Spark) having many > partitions, > the class FileOutputCommiter process thousands of dirs/files to rename with a > single Thread. This is performance issue, caused by lot of waits on > FileStystem storage operations. > I propose that above a configurable threshold (default=3, configurable via > property 'mapreduce.fileoutputcommitter.parallel.threshold'), the class > FileOutputCommiter process the list of files to rename using parallel > threads, using the default jvm ExecutorService (ForkJoinPool.commonPool()) > See Pull-Request: > [https://github.com/apache/hadoop/pull/6378|https://github.com/apache/hadoop/pull/6378] > Notice that sub-class instances of FileOutputCommiter are supposed to be > created at runtime dependending of a configurable property > ([https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PathOutputCommitterFactory.java|PathOutputCommitterFactory.java]). > But for example in Parquet + Spark, this is buggy and can not be changed at > runtime. > There is an ongoing Jira and PR to fix it in Parquet + Spark: > [https://issues.apache.org/jira/browse/PARQUET-2416|https://issues.apache.org/jira/browse/PARQUET-2416] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7465) performance problem in FileOutputCommiter for big list processed by single thread
[ https://issues.apache.org/jira/browse/MAPREDUCE-7465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801584#comment-17801584 ] Steve Loughran commented on MAPREDUCE-7465: --- I understand your pain here, but am reluctant to do this for a few reasons * fear of going near a complicated and critical piece of code. I am scared of it. It's got two co-recursive algorithms and is a critical workflow. * experience of an internal release with this against abfs. it's not enough: hits throttling and not resilient to failures. * non-atomic task commit with google gcs * dir tree cleanup scale issues with abfs and gcs The good news: the MAPREDUCE-7341 manifest committer addresses all of this * correct and performant with hdfs, gcs and abfs * it rate limits (and resiliently) renames against abfs * because it does the task attempt dir scan on task commit, there's no need for parallel treewalks in job commit. * parallel task attempt cleanup for filesystems with O(files) dir delete * the _SUCCESS file is json stats file with timings; can optionally be saved elsewhere too. Your PR isn't going to be get to any release which doesn't have the intermediate manifest committer in it. Now, I can see you are making changes to parquet to work better here; this is lovely. I'll review/comment those. I think a key one is for parquet and spark to both lower their expectations of committer types. Spark's hadoop-cloud module does have the delegation class you already need, it's documented in https://spark.apache.org/docs/latest/cloud-integration.html this means you should be able to switch to the manifest committer *today* if your hadoop mapreduce library supports it. Yes, it does work with HDFS too, it's just not tested as rigorously...we have gcs and abfs using this as the default. What I will do is do a pr up with our own internal change...this isn't something we enable as the manifest committer is so much better. but you can compare it with yours, including the tests, and know that we have at least done some NFQE testing with it. > performance problem in FileOutputCommiter for big list processed by single > thread > -- > > Key: MAPREDUCE-7465 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7465 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance >Affects Versions: 3.2.3, 3.3.2, 3.2.4, 3.3.5, 3.3.3, 3.3.4, 3.3.6 >Reporter: Arnaud Nauwynck >Priority: Minor > Labels: pull-request-available > > when commiting a big hadoop job (for example via Spark) having many > partitions, > the class FileOutputCommiter process thousands of dirs/files to rename with a > single Thread. This is performance issue, caused by lot of waits on > FileStystem storage operations. > I propose that above a configurable threshold (default=3, configurable via > property 'mapreduce.fileoutputcommitter.parallel.threshold'), the class > FileOutputCommiter process the list of files to rename using parallel > threads, using the default jvm ExecutorService (ForkJoinPool.commonPool()) > See Pull-Request: > [https://github.com/apache/hadoop/pull/6378|https://github.com/apache/hadoop/pull/6378] > Notice that sub-class instances of FileOutputCommiter are supposed to be > created at runtime dependending of a configurable property > ([https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PathOutputCommitterFactory.java|PathOutputCommitterFactory.java]). > But for example in Parquet + Spark, this is buggy and can not be changed at > runtime. > There is an ongoing Jira and PR to fix it in Parquet + Spark: > [https://issues.apache.org/jira/browse/PARQUET-2416|https://issues.apache.org/jira/browse/PARQUET-2416] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-7459) Sort strings to fix flaky test
[
https://issues.apache.org/jira/browse/MAPREDUCE-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran moved HADOOP-18947 to MAPREDUCE-7459:
Component/s: test
(was: test)
Fix Version/s: 3.3.6
(was: 3.3.6)
Key: MAPREDUCE-7459 (was: HADOOP-18947)
Affects Version/s: 3.3.6
(was: 3.3.6)
Project: Hadoop Map/Reduce (was: Hadoop Common)
> Sort strings to fix flaky test
> --
>
> Key: MAPREDUCE-7459
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7459
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 3.3.6
> Environment: Java version: openjdk 11.0.20.1
> Maven version: Apache Maven 3.6.3
>Reporter: Rajiv Ramachandran
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.6
>
>
> The test
> {{_org.apache.hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter#testHumanPrinterAll_}}
> can fail due to flakiness. These flakiness occurs because the test utilizes
> Hashmaps values and converts the values to string to perform the comparision
> and the order of the objects returned may not be necessarily maintained.
> The stack trace is as follows:
> testHumanPrinterAll(org.apache.hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter)
> Time elapsed: 0.297 s <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...8501754_0001_m_0[7 6-Oct-2011 19:15:09 6-Oct-2011
> 19:15:16 (7sec)
> SUCCEEDED MAP task list for job_1317928501754_0001
> TaskId StartTime FinishTime Error InputSplits
>
> task_1317928501754_0001_m_06 6-Oct-2011 19:15:08 6-Oct-2011
> 19:15:14 (6sec)
> ...
> /tasklog?attemptid=attempt_1317928501754_0001_m_03]_1
> REDUCE task list...> but was:<...8501754_0001_m_0[5 6-Oct-2011
> 19:15:07 6-Oct-2011 19:15:12 (5sec)
> SUCCEEDED MAP task list for job_1317928501754_0001
> TaskId StartTime FinishTime Error InputSplits
>
> task_1317928501754_0001_m_06 6-Oct-2011 19:15:08 6-Oct-2011
> 19:15:14 (6sec)
> SUCCEEDED MAP task list for job_1317928501754_0001
> TaskId StartTime FinishTime Error InputSplits
>
> task_1317928501754_0001_m_04 6-Oct-2011 19:15:06 6-Oct-2011
> 19:15:10 (4sec)
> SUCCEEDED MAP task list for job_1317928501754_0001
> TaskId StartTime FinishTime Error InputSplits
>
> task_1317928501754_0001_m_07 6-Oct-2011 19:15:09 6-Oct-2011
> 19:15:16 (7sec)
> ...
> /tasklog?attemptid=attempt_1317928501754_0001_m_06]_1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7449) Add add-opens flag to container launch commands on JDK17 nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7449: -- Fix Version/s: 3.3.9 Resolution: Fixed Status: Resolved (was: Patch Available) > Add add-opens flag to container launch commands on JDK17 nodes > -- > > Key: MAPREDUCE-7449 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7449 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mr-am, mrv2 >Affects Versions: 3.4.0, 3.3.7 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > To allow containers to launch on JDK17 nodes the add-opens flag should be > added to the container launch commands if the node has JDK17+, and it > shouldn't on previous JDKs. This behaviour should be configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7449) Add add-opens flag to container launch commands on JDK17 nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760010#comment-17760010 ] Steve Loughran commented on MAPREDUCE-7449: --- HADOOP-18862 should be fixed by this; will test. can we get this cherrypicked to branch-3.3 so the next release we push out works? > Add add-opens flag to container launch commands on JDK17 nodes > -- > > Key: MAPREDUCE-7449 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7449 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mr-am, mrv2 >Affects Versions: 3.4.0 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > To allow containers to launch on JDK17 nodes the add-opens flag should be > added to the container launch commands if the node has JDK17+, and it > shouldn't on previous JDKs. This behaviour should be configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7451) review TrackerDistributedCacheManager.checkPermissionOfOther
[ https://issues.apache.org/jira/browse/MAPREDUCE-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7451. --- Resolution: Won't Fix The class TrackerDistributedCacheManager only exists in hadoop releases <= 1.2. no need to look at it > review TrackerDistributedCacheManager.checkPermissionOfOther > > > Key: MAPREDUCE-7451 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7451 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0, 1.2.1 >Reporter: Yiheng Cao >Priority: Major > > TrackerDistributedCacheManager.checkPermissionOfOther() doesn't seem to work > reliably -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7451) review TrackerDistributedCacheManager.checkPermissionOfOther
[ https://issues.apache.org/jira/browse/MAPREDUCE-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7451: -- Affects Version/s: 1.2.1 0.22.0 > review TrackerDistributedCacheManager.checkPermissionOfOther > > > Key: MAPREDUCE-7451 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7451 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0, 1.2.1 >Reporter: Yiheng Cao >Priority: Major > > TrackerDistributedCacheManager.checkPermissionOfOther() doesn't seem to work > reliably -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7452) ManifestCommitter to support / as a destination
Steve Loughran created MAPREDUCE-7452: - Summary: ManifestCommitter to support / as a destination Key: MAPREDUCE-7452 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7452 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 3.3.6 Reporter: Steve Loughran you can't commit work to the root of an object store through the manifest committer, as it will fail if the destination path exists, which always holds for root. proposed * check for dest / in job setup; if the path is not root, use createNewDirectory() as today * if the path is root, delete all children but not the dir. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7451) review TrackerDistributedCacheManager.checkPermissionOfOther
[
https://issues.apache.org/jira/browse/MAPREDUCE-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7451:
--
Summary: review TrackerDistributedCacheManager.checkPermissionOfOther
(was: Security Vulnerability - Action Required: “Incorrect Permission
Assignment for Critical Resource” vulnerability in the newest version of hadoop)
> review TrackerDistributedCacheManager.checkPermissionOfOther
>
>
> Key: MAPREDUCE-7451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7451
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Yiheng Cao
>Priority: Major
>
> I think the method
> {{org.apache.hadoop.filecache.TrackerDistributedCacheManager.checkPermissionOfOther(FileSystem
> fs, Path path, FsAction action)}} may have an “Incorrect Permission
> Assignment for Critical Resource”vulnerability which is vulnerable in the
> newest version of hadoop. It shares similarities to a recent CVE disclosure
> _CVE-2017-3166_ in the same project _"apache/hadoop"_ project.
> The vulnerability is present in the class
> org.apache.hadoop.filecache.TrackerDistributedCacheManager of method
> checkPermissionOfOther(FileSystem fs, Path path, FsAction action), which is
> responsible for Checking whether the file system object (FileSystem) at the
> specified path has additional user permissions for the specified
> operation(action). {*}But t{*}{*}he check snippet is similar to the
> vulnerable snippet for CVE-2017-3166{*} and may have the same consequence as
> CVE-2017-3166: {*}a file in an encryption zone with access permissions will
> be stored in a world-readable location and can be freely shared with any
> application that requests the file to be localized{*}. Therefore, maybe you
> need to fix the vulnerability with much the same fix code as the
> CVE-2017-3166 patch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7451) review TrackerDistributedCacheManager.checkPermissionOfOther
[
https://issues.apache.org/jira/browse/MAPREDUCE-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7451:
--
Description: TrackerDistributedCacheManager.checkPermissionOfOther()
doesn't seem to work reliably (was: I think the method
{{org.apache.hadoop.filecache.TrackerDistributedCacheManager.checkPermissionOfOther(FileSystem
fs, Path path, FsAction action)}} may have an “Incorrect Permission Assignment
for Critical Resource”vulnerability which is vulnerable in the newest version
of hadoop. It shares similarities to a recent CVE disclosure _CVE-2017-3166_ in
the same project _"apache/hadoop"_ project.
The vulnerability is present in the class
org.apache.hadoop.filecache.TrackerDistributedCacheManager of method
checkPermissionOfOther(FileSystem fs, Path path, FsAction action), which is
responsible for Checking whether the file system object (FileSystem) at the
specified path has additional user permissions for the specified
operation(action). {*}But t{*}{*}he check snippet is similar to the vulnerable
snippet for CVE-2017-3166{*} and may have the same consequence as
CVE-2017-3166: {*}a file in an encryption zone with access permissions will
be stored in a world-readable location and can be freely shared with any
application that requests the file to be localized{*}. Therefore, maybe you
need to fix the vulnerability with much the same fix code as the CVE-2017-3166
patch. )
> review TrackerDistributedCacheManager.checkPermissionOfOther
>
>
> Key: MAPREDUCE-7451
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7451
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Yiheng Cao
>Priority: Major
>
> TrackerDistributedCacheManager.checkPermissionOfOther() doesn't seem to work
> reliably
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7448) Inconsistent Behavior for FileOutputCommitter V1 to commit successfully many times
[
https://issues.apache.org/jira/browse/MAPREDUCE-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17748946#comment-17748946
]
Steve Loughran commented on MAPREDUCE-7448:
---
process: can you tag with the version you are reporting on?
After a quick glance at the source, I have to agree. Sort of.
job IDs aren't used in the paths, so the output of attempt 0 goes into
_temporary/0, attempt 1 into _temporary/1, etc.
If a job with a different job ID is run on the same final destination path *and
you have configured that job to add files to the existing path* then the next
job will have the same path. It may pick up the existing files -except if that
previous job commit succeeded, the files will already have been renamed. Only
the new files from the new job should be picked up.
Even so, it's not particularly safe. Using job ID under _temporary would appear
to be a solution, but it requires job IDs to be unique, which
HADOOP-17318/SPARK-33230 show is not already true.
And we are scared of making changes to that committer because it is such a
critical piece of code.
The disabling cleanup option is only there to speed up cleanup on GCS storage
with O(dir) deletion -I'd recommend a documentation patch warning about.
Now
# which version of hadoop are you using?
# what filesystem? HDFS or something else?
If you are on hadoop 3.3.5, can you try the manifest committer instead? it does
work for HDFS, even if it is optimised for cloud stores where list, rename and
delete are a lot slower.
> Inconsistent Behavior for FileOutputCommitter V1 to commit successfully many
> times
> --
>
> Key: MAPREDUCE-7448
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7448
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: ConfX
>Priority: Critical
> Attachments: reproduce.sh
>
>
> h2. What happened
> I turned on {{mapreduce.fileoutputcommitter.cleanup.skipped=true}} and then
> the version 1 of {{FileOutputCommitter}} can commit several times, which is
> unexpected.
> h2. Where's the problem
> In {{{}FileOutputCommitter.commitJobInternal{}}},
> {noformat}
> if (algorithmVersion == 1) {
> for (FileStatus stat: getAllCommittedTaskPaths(context)) {
> mergePaths(fs, stat, finalOutput, context);
> }
> } if (skipCleanup) {
> LOG.info("Skip cleanup the _temporary folders under job's output " +
> "directory in commitJob.");
> ...{noformat}
> Here if we skip cleanup, the _temporary folder would not be deleted and the
> _SUCCESS file would also not be created, which cause the {{mergePaths}} next
> time to not fail.
> h2. How to reproduce
> # set {{{}mapreduce.fileoutputcommitter.cleanup.skipped{}}}={{{}true{}}}
> # run
> {{org.apache.hadoop.mapred.TestFileOutputCommitter#testCommitterWithDuplicatedCommitV1}}
> you should observe
> {noformat}
> java.lang.AssertionError: Duplicate commit successful: wrong behavior for
> version 1.
> at org.junit.Assert.fail(Assert.java:89)
> at
> org.apache.hadoop.mapred.TestFileOutputCommitter.testCommitterWithDuplicatedCommitInternal(TestFileOutputCommitter.java:295)
> at
> org.apache.hadoop.mapred.TestFileOutputCommitter.testCommitterWithDuplicatedCommitV1(TestFileOutputCommitter.java:269){noformat}
> For an easy reproduction, run the reproduce.sh in the attachment.
> We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7432) Make Manifest Committer the default for abfs and gcs
[ https://issues.apache.org/jira/browse/MAPREDUCE-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7432. --- Fix Version/s: 3.4.0 3.3.9 Release Note: By default, the mapreduce manifest committer is used for jobs working with abfs and gcs.. Hadoop mapreduce jobs will pick this up automatically; for Spark it is a bit complicated: read the docs to see the steps required. Resolution: Fixed > Make Manifest Committer the default for abfs and gcs > > > Key: MAPREDUCE-7432 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7432 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Switch to the manifest committer as default for abfs and gcs > * abfs: needed for performance, scale and resilience under some failure modes > * gcs: provides correctness through atomic task commit and better job commit > performance -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7435) ManifestCommitter OOM on azure job
[
https://issues.apache.org/jira/browse/MAPREDUCE-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved MAPREDUCE-7435.
---
Fix Version/s: 3.4.0
3.3.9
Resolution: Fixed
> ManifestCommitter OOM on azure job
> --
>
> Key: MAPREDUCE-7435
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7435
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.3.5
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> I've got some reports of spark jobs OOM if the manifest committer is used
> through abfs.
> either the manifests are using too much memory, or something is not working
> with azure stream memory use (or both).
> before proposing a solution, first step should be to write a test to load
> many, many manifests, each with lots of dirs and files to see what breaks.
> note: we did have OOM issues with the s3a committer, on teragen but those
> structures have to include every etag of every block, so the manifest size is
> O(blocks); the new committer is O(files + dirs).
> {code}
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.readOneBlock(AbfsInputStream.java:314)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.read(AbfsInputStream.java:267)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:539)
> at
> com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:133)
> at
> com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:256)
> at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1656)
> at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1085)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3585)
> at
> org.apache.hadoop.util.JsonSerialization.fromJsonStream(JsonSerialization.java:164)
> at org.apache.hadoop.util.JsonSerialization.load(JsonSerialization.java:279)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.TaskManifest.load(TaskManifest.java:361)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem.loadTaskManifest(ManifestStoreOperationsThroughFileSystem.java:133)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.lambda$loadManifest$6(AbstractJobOrTaskStage.java:493)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage$$Lambda$231/1813048085.apply(Unknown
> Source)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding$$Lambda$217/489150849.apply(Unknown
> Source)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.loadManifest(AbstractJobOrTaskStage.java:492)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.LoadManifestsStage.fetchTaskManifest(LoadManifestsStage.java:170)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.LoadManifestsStage.processOneManifest(LoadManifestsStage.java:138)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.LoadManifestsStage$$Lambda$229/137752948.run(Unknown
> Source)
> at
> org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
> at
> org.apache.hadoop.util.functional.TaskPool$Builder$$Lambda$230/467893357.run(Unknown
> Source)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7437) MR Fetcher class to use an AtomicInteger to generate IDs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7437: -- Summary: MR Fetcher class to use an AtomicInteger to generate IDs. (was: spotbugs complaining about .Fetcher's update of a nonatomic static counter) > MR Fetcher class to use an AtomicInteger to generate IDs. > - > > Key: MAPREDUCE-7437 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7437 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, client >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > I'm having to do this to get MAPREDUCE-7435 through the build; spotbugs is > complaining about the Fetcher constructor incrementing a non-static shared > counter. Which is true, just odd it has only just surfaced. > going to fix as a standalone patch but include that in the commit chain of > that PR too -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7437) spotbugs complaining about .Fetcher's update of a nonatomic static counter
[ https://issues.apache.org/jira/browse/MAPREDUCE-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7437. --- Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed > spotbugs complaining about .Fetcher's update of a nonatomic static counter > -- > > Key: MAPREDUCE-7437 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7437 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, client >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > I'm having to do this to get MAPREDUCE-7435 through the build; spotbugs is > complaining about the Fetcher constructor incrementing a non-static shared > counter. Which is true, just odd it has only just surfaced. > going to fix as a standalone patch but include that in the commit chain of > that PR too -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7437) spotbugs complaining about .Fetcher's update of a nonatomic static counter
[
https://issues.apache.org/jira/browse/MAPREDUCE-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715024#comment-17715024
]
Steve Loughran commented on MAPREDUCE-7437:
---
{code}
CodeWarning
ST Write to static field
org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new
org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID,
ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics,
ExceptionReporter, SecretKey)
Bug type ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD (click for details)
In class org.apache.hadoop.mapreduce.task.reduce.Fetcher
In method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf,
TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter,
ShuffleClientMetrics, ExceptionReporter, SecretKey)
Field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId
At Fetcher.java:[line 120]
{code}
> spotbugs complaining about .Fetcher's update of a nonatomic static counter
> --
>
> Key: MAPREDUCE-7437
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7437
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, client
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> I'm having to do this to get MAPREDUCE-7435 through the build; spotbugs is
> complaining about the Fetcher constructor incrementing a non-static shared
> counter. Which is true, just odd it has only just surfaced.
> going to fix as a standalone patch but include that in the commit chain of
> that PR too
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7437) spotbugs complaining about .Fetcher's update of a nonatomic static counter
Steve Loughran created MAPREDUCE-7437: - Summary: spotbugs complaining about .Fetcher's update of a nonatomic static counter Key: MAPREDUCE-7437 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7437 Project: Hadoop Map/Reduce Issue Type: Bug Components: build, client Affects Versions: 3.4.0 Reporter: Steve Loughran Assignee: Steve Loughran I'm having to do this to get MAPREDUCE-7435 through the build; spotbugs is complaining about the Fetcher constructor incrementing a non-static shared counter. Which is true, just odd it has only just surfaced. going to fix as a standalone patch but include that in the commit chain of that PR too -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7435) ManifestCommitter OOM on azure job
[
https://issues.apache.org/jira/browse/MAPREDUCE-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707960#comment-17707960
]
Steve Loughran commented on MAPREDUCE-7435:
---
comments of HADOOP-18650 also apply, especially collection of stats on manifest
file sizes and number of renames/dirs per manifest
> ManifestCommitter OOM on azure job
> --
>
> Key: MAPREDUCE-7435
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7435
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.3.5
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> I've got some reports of spark jobs OOM if the manifest committer is used
> through abfs.
> either the manifests are using too much memory, or something is not working
> with azure stream memory use (or both).
> before proposing a solution, first step should be to write a test to load
> many, many manifests, each with lots of dirs and files to see what breaks.
> note: we did have OOM issues with the s3a committer, on teragen but those
> structures have to include every etag of every block, so the manifest size is
> O(blocks); the new committer is O(files + dirs).
> {code}
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.readOneBlock(AbfsInputStream.java:314)
> at
> org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.read(AbfsInputStream.java:267)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:539)
> at
> com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:133)
> at
> com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:256)
> at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1656)
> at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1085)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3585)
> at
> org.apache.hadoop.util.JsonSerialization.fromJsonStream(JsonSerialization.java:164)
> at org.apache.hadoop.util.JsonSerialization.load(JsonSerialization.java:279)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.TaskManifest.load(TaskManifest.java:361)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem.loadTaskManifest(ManifestStoreOperationsThroughFileSystem.java:133)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.lambda$loadManifest$6(AbstractJobOrTaskStage.java:493)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage$$Lambda$231/1813048085.apply(Unknown
> Source)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding$$Lambda$217/489150849.apply(Unknown
> Source)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.loadManifest(AbstractJobOrTaskStage.java:492)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.LoadManifestsStage.fetchTaskManifest(LoadManifestsStage.java:170)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.LoadManifestsStage.processOneManifest(LoadManifestsStage.java:138)
> at
> org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.LoadManifestsStage$$Lambda$229/137752948.run(Unknown
> Source)
> at
> org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
> at
> org.apache.hadoop.util.functional.TaskPool$Builder$$Lambda$230/467893357.run(Unknown
> Source)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7435) ManifestCommitter OOM on azure job
Steve Loughran created MAPREDUCE-7435:
-
Summary: ManifestCommitter OOM on azure job
Key: MAPREDUCE-7435
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7435
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.3.5
Reporter: Steve Loughran
Assignee: Steve Loughran
I've got some reports of spark jobs OOM if the manifest committer is used
through abfs.
either the manifests are using too much memory, or something is not working
with azure stream memory use (or both).
before proposing a solution, first step should be to write a test to load many,
many manifests, each with lots of dirs and files to see what breaks.
note: we did have OOM issues with the s3a committer, on teragen but those
structures have to include every etag of every block, so the manifest size is
O(blocks); the new committer is O(files + dirs).
{code}
java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.readOneBlock(AbfsInputStream.java:314)
at
org.apache.hadoop.fs.azurebfs.services.AbfsInputStream.read(AbfsInputStream.java:267)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:539)
at
com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:133)
at
com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:256)
at com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1656)
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1085)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3585)
at
org.apache.hadoop.util.JsonSerialization.fromJsonStream(JsonSerialization.java:164)
at org.apache.hadoop.util.JsonSerialization.load(JsonSerialization.java:279)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.TaskManifest.load(TaskManifest.java:361)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem.loadTaskManifest(ManifestStoreOperationsThroughFileSystem.java:133)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.lambda$loadManifest$6(AbstractJobOrTaskStage.java:493)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage$$Lambda$231/1813048085.apply(Unknown
Source)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding$$Lambda$217/489150849.apply(Unknown
Source)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.loadManifest(AbstractJobOrTaskStage.java:492)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.LoadManifestsStage.fetchTaskManifest(LoadManifestsStage.java:170)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.LoadManifestsStage.processOneManifest(LoadManifestsStage.java:138)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.LoadManifestsStage$$Lambda$229/137752948.run(Unknown
Source)
at
org.apache.hadoop.util.functional.TaskPool$Builder.lambda$runParallel$0(TaskPool.java:410)
at
org.apache.hadoop.util.functional.TaskPool$Builder$$Lambda$230/467893357.run(Unknown
Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7432) Make Manifest Committer the default for abfs and gcs
[ https://issues.apache.org/jira/browse/MAPREDUCE-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned MAPREDUCE-7432: - Assignee: Steve Loughran > Make Manifest Committer the default for abfs and gcs > > > Key: MAPREDUCE-7432 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7432 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Switch to the manifest committer as default for abfs and gcs > * abfs: needed for performance, scale and resilience under some failure modes > * gcs: provides correctness through atomic task commit and better job commit > performance -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7432) Make Manifest Committer the default for abfs and gcs
Steve Loughran created MAPREDUCE-7432: - Summary: Make Manifest Committer the default for abfs and gcs Key: MAPREDUCE-7432 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7432 Project: Hadoop Map/Reduce Issue Type: New Feature Components: client Affects Versions: 3.3.5 Reporter: Steve Loughran Switch to the manifest committer as default for abfs and gcs * abfs: needed for performance, scale and resilience under some failure modes * gcs: provides correctness through atomic task commit and better job commit performance -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7375) JobSubmissionFiles don't set right permission after mkdirs
[
https://issues.apache.org/jira/browse/MAPREDUCE-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7375:
--
Fix Version/s: 3.3.5
(was: 3.3.9)
> JobSubmissionFiles don't set right permission after mkdirs
> --
>
> Key: MAPREDUCE-7375
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7375
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.3.2
>Reporter: Zhang Dongsheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.2.5
>
> Attachments: MAPREDUCE-7375.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> JobSubmissionFiles provide getStagingDir to get Staging Directory.If
> stagingArea missing, method will create new directory with this.
> {quote}fs.mkdirs(stagingArea, new FsPermission(JOB_DIR_PERMISSION));{quote}
> It seems create new directory with JOB_DIR_PERMISSION,but this permission
> will be apply by umask.If umask too strict , this permission may be 000(if
> umask is 700).So we should change permission after create.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7428) Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648997#comment-17648997 ] Steve Loughran commented on MAPREDUCE-7428: --- i'm doing a full revert. don't want to have things being skipped. we've already just been burned by javadoc > Fix failures related to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > --- > > Key: MAPREDUCE-7428 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7428 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > > Few test are getting failed due to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > [https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1071/testReport/] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7428) Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648995#comment-17648995 ] Steve Loughran commented on MAPREDUCE-7428: --- the brute force action is to roll back all the mr-junit patches and conclude "harder than it looks", maybe to the extent of creating a new dev branch and wiring jenkins up to it for full transitive coverage? > Fix failures related to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > --- > > Key: MAPREDUCE-7428 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7428 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > > Few test are getting failed due to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > [https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1071/testReport/] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7428) Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648768#comment-17648768 ] Steve Loughran commented on MAPREDUCE-7428: --- revert? > Fix failures related to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > --- > > Key: MAPREDUCE-7428 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7428 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > > Few test are getting failed due to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > [https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1071/testReport/] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7428) Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648115#comment-17648115 ] Steve Loughran commented on MAPREDUCE-7428: --- [~groot] to repeate my question: which patch caused the failure *and why didn't yetus notice* > Fix failures related to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > --- > > Key: MAPREDUCE-7428 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7428 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > > Few test are getting failed due to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > [https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1071/testReport/] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7428) Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7428. --- Fix Version/s: 3.4.0 Resolution: Fixed > Fix failures related to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > --- > > Key: MAPREDUCE-7428 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7428 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > > Few test are getting failed due to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > [https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1071/testReport/] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7428) Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7428: -- Priority: Critical (was: Major) > Fix failures related to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > --- > > Key: MAPREDUCE-7428 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7428 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Critical > Labels: pull-request-available > > Few test are getting failed due to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > [https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1071/testReport/] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7422) Upgrade Junit 4 to 5 in hadoop-mapreduce-examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-7422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17647080#comment-17647080 ] Steve Loughran commented on MAPREDUCE-7422: --- I'm assuming this is the cause of MAPREDUCE-7428. > Upgrade Junit 4 to 5 in hadoop-mapreduce-examples > - > > Key: MAPREDUCE-7422 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7422 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: test >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Upgrade Junit 4 to 5 in hadoop-mapreduce-examples -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7428) Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17646540#comment-17646540 ] Steve Loughran commented on MAPREDUCE-7428: --- 1. what is the jira which caused this? add a link and a backwards ref in the commit message 2. how did this regression get past yetus? > Fix failures related to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > --- > > Key: MAPREDUCE-7428 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7428 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Few test are getting failed due to Junit 4 to Junit 5 upgrade in > org.apache.hadoop.mapreduce.v2.app.webapp > [https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64/1071/testReport/] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7401) Optimize liststatus for better performance by using recursive listing
[ https://issues.apache.org/jira/browse/MAPREDUCE-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7401. --- Resolution: Won't Fix > Optimize liststatus for better performance by using recursive listing > - > > Key: MAPREDUCE-7401 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7401 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 3.3.3 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This change adds recursive listing APIs to FileSystem. The purpose is to > enable different FileSystem implementations optimize on the listStatus calls > if they can. Default implementation is provided for normal FileSystem > implementation which does level by level listing for each directory. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7401) Optimize liststatus for better performance by using recursive listing
[ https://issues.apache.org/jira/browse/MAPREDUCE-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17640794#comment-17640794 ] Steve Loughran commented on MAPREDUCE-7401: --- the current pr isn't going in for the reasons described in the pr. listFiles(path, recursive) is the existing api...the challenge is to allow apps to use it without breaking the current apis -which is really, really hard -1, sorry. Do not go near this unless you can show that the current `listFiles(path, recursive)' is inadequate. Which I do not believe it is. If you can make the case that it doesn't change it then you have to look very closely at the Javadocs at the top of FileSystem and any recent changes to the API to see how they are managed. Vectored IO for example. also look at HADOOP-16898 and HADOOP-16898 to see their listing changes including my unhappiness about something going in without more publicity across the different teams. Any change in that API is public facing and has to be maintained forever. It needs to be supported effectively in HDFS and in cloud storage. That means you're going to have to do a full api specification, write contract tests, implement those contact tests on in hadoop-aws and azure, and ideally anywhere else (google gcs). then make sure that you don't break the external libs named in the javadocs. Assume that I will automatically veto any new list method returning an array. It hits scale problems on HDFS -lock duration, size of responses to marshall- and prevents us doing things in the object stores including prefetching, IOStatistics collection and supporting close(). Also using builder APIs and returning a CompletableFuture. Look at the s3a and abfs listing code to see how implement listFiles, and the s3a and manifest I committed to see how they are effectively used. we kick off operations (treewalk, file loading) while waiting for next page of responses to come in, ideally swallowing the entire latency of each list call. Note also that because listFiles only returns files, not directories, we can do O(files/page size) deep list calls against s3. If the justification is that we need path filtering, see HADOOP-16673 _Add filter parameter to FileSystem>>listFiles_ to see why that doesn't work in cloud and hence closed as WONTFIX. I think a more manageable focus of this work would be to see how FileInputFormat could be speeded up by using the existing APIs, I am at with all work done knowing that many external libraries subclass that. For example, Parquet, Avro and ORC. Any incompatible change will stop them upgrading and we cannot do that. Am I being very negative here? Yes I am. If you do want to change the Apis then you need to start talking about it on the HDFS and common lists, show that it delivers tangible benefit on-prem and in cloud, and undertake the extensive piece of work needed to implement in the primary cloud stores to show it is performant. > Optimize liststatus for better performance by using recursive listing > - > > Key: MAPREDUCE-7401 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7401 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 3.3.3 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This change adds recursive listing APIs to FileSystem. The purpose is to > enable different FileSystem implementations optimize on the listStatus calls > if they can. Default implementation is provided for normal FileSystem > implementation which does level by level listing for each directory. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7401) Optimize liststatus for better performance by using recursive listing
[ https://issues.apache.org/jira/browse/MAPREDUCE-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7401: -- Target Version/s: (was: 3.3.5) > Optimize liststatus for better performance by using recursive listing > - > > Key: MAPREDUCE-7401 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7401 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 3.3.3 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This change adds recursive listing APIs to FileSystem. The purpose is to > enable different FileSystem implementations optimize on the listStatus calls > if they can. Default implementation is provided for normal FileSystem > implementation which does level by level listing for each directory. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7386) Maven parallel builds (skipping tests) fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-7386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved MAPREDUCE-7386.
---
Fix Version/s: 3.4.0
Resolution: Fixed
in trunk, backport once we are happy that it is stable
> Maven parallel builds (skipping tests) fail
> ---
>
> Key: MAPREDUCE-7386
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7386
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0, 3.3.5
> Environment: The problem occurred while using the Hadoop development
> environment (Ubuntu)
>Reporter: Steve Vaughan
>Assignee: Steve Vaughan
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Running a parallel build fails during assembly with the following error when
> running either package or install:
> {code:java}
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-assembly-plugin:2.4:single
> (package-mapreduce) on project hadoop-mapreduce: Failed to create assembly:
> Artifact: org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.4.0-SNAPSHOT
> (included by module) does not have an artifact with a file. Please ensure the
> package phase is run before the assembly is generated. {code}
> {code:java}
> Caused by: org.apache.maven.plugin.MojoExecutionException: Failed to create
> assembly: Artifact:
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.4.0-SNAPSHOT (included
> by module) does not have an artifact with a file. Please ensure the package
> phase is run before the assembly is generated. {code}
> The command executed was:
> {code:java}
> $ mvn -nsu clean install -Pdist,native -DskipTests -Dtar
> -Dmaven.javadoc.skip=true -T 2C {code}
> Adding dependencies to the assembly plugin configuration addresses the issue
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7411) Use secure XML parser utils in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624459#comment-17624459 ] Steve Loughran commented on MAPREDUCE-7411: --- ok; changetd to the newer a/c > Use secure XML parser utils in MapReduce > > > Key: MAPREDUCE-7411 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7411 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Uptake of HADOOP-18469 > If anyone is landing on this page following any security scanner alert, know > that there is no known issue here, just a centralisation of all construction > of XML parsers with lockdown of all the features. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7411) Use secure XML parser utils in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned MAPREDUCE-7411: - Assignee: PJ Fanning (was: PJ Fanning) > Use secure XML parser utils in MapReduce > > > Key: MAPREDUCE-7411 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7411 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Uptake of HADOOP-18469 > If anyone is landing on this page following any security scanner alert, know > that there is no known issue here, just a centralisation of all construction > of XML parsers with lockdown of all the features. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7411) Use secure XML parser utils in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624348#comment-17624348 ] Steve Loughran commented on MAPREDUCE-7411: --- [~fanningpj] assigned to you after adding you to the right role in mapreduce. did i do it for the right account of yours? > Use secure XML parser utils in MapReduce > > > Key: MAPREDUCE-7411 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7411 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Uptake of HADOOP-18469 > If anyone is landing on this page following any security scanner alert, know > that there is no known issue here, just a centralisation of all construction > of XML parsers with lockdown of all the features. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7411) Use secure XML parser utils in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned MAPREDUCE-7411: - Assignee: PJ Fanning > Use secure XML parser utils in MapReduce > > > Key: MAPREDUCE-7411 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7411 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Uptake of HADOOP-18469 > If anyone is landing on this page following any security scanner alert, know > that there is no known issue here, just a centralisation of all construction > of XML parsers with lockdown of all the features. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7411) Use secure XML parser utils in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7411: -- Description: Uptake of HADOOP-18469 If anyone is landing on this page following any security scanner alert, know that there is no known issue here, just a centralisation of all construction of XML parsers with lockdown of all the features. was:Uptake of HADOOP-18469 > Use secure XML parser utils in MapReduce > > > Key: MAPREDUCE-7411 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7411 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Uptake of HADOOP-18469 > If anyone is landing on this page following any security scanner alert, know > that there is no known issue here, just a centralisation of all construction > of XML parsers with lockdown of all the features. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7411) Use secure XML parser utils in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7411. --- Fix Version/s: 3.4.0 3.3.5 Resolution: Fixed merged back to branch-3.3.5 > Use secure XML parser utils in MapReduce > > > Key: MAPREDUCE-7411 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7411 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > > Uptake of HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7411) Use secure XML parser utils in MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7411: -- Summary: Use secure XML parser utils in MapReduce (was: Use secure XML parser utils) > Use secure XML parser utils in MapReduce > > > Key: MAPREDUCE-7411 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7411 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > > Uptake of HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7403) Support spark dynamic partitioning in the Manifest Committer
[ https://issues.apache.org/jira/browse/MAPREDUCE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7403. --- Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed hadoop side is in; needs spark to match > Support spark dynamic partitioning in the Manifest Committer > > > Key: MAPREDUCE-7403 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7403 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.3.9 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Currently the spark integration with PathOutputCommitters rejects attempt to > instantiate them if dynamic partitioning is enabled. That is because the > spark partitioning code assumes that > # file rename works as a fast and safe commit algorithm > # the working directory is in the same FS as the final directory > Assumption 1 doesn't hold on s3a, and #2 isn't true for the staging > committers. > The new abfs/gcs manifest committer and the target stores do meet both > requirements. So we no longer need to reject the operation, provided the > spark side binding-code can can identify when all is good. > Proposed: add a new hasCapability() probe which, if, a committer implements > StreamCapabilities can be used to see if the committer will work. > ManifestCommitter will declare that it holds. As the API has existed since > 2.10, it will be immediately available. > spark's PathOutputCommitProtocol to query the committer in setupCommitter, > and fail if dynamicPartitionOverwrite is requested but not available. > BindingParquetOutputCommitter to implement and forward > StreamCapabilities.hasCapability. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7403) Support spark dynamic partitioning in the Manifest Committer
Steve Loughran created MAPREDUCE-7403: - Summary: Support spark dynamic partitioning in the Manifest Committer Key: MAPREDUCE-7403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7403 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 3.3.9 Reporter: Steve Loughran Assignee: Steve Loughran Currently the spark integration with PathOutputCommitters rejects attempt to instantiate them if dynamic partitioning is enabled. That is because the spark partitioning code assumes that # file rename works as a fast and safe commit algorithm # the working directory is in the same FS as the final directory Assumption 1 doesn't hold on s3a, and #2 isn't true for the staging committers. The new abfs/gcs manifest committer and the target stores do meet both requirements. So we no longer need to reject the operation, provided the spark side binding-code can can identify when all is good. Proposed: add a new hasCapability() probe which, if, a committer implements StreamCapabilities can be used to see if the committer will work. ManifestCommitter will declare that it holds. As the API has existed since 2.10, it will be immediately available. spark's PathOutputCommitProtocol to query the committer in setupCommitter, and fail if dynamicPartitionOverwrite is requested but not available. BindingParquetOutputCommitter to implement and forward StreamCapabilities.hasCapability. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7398) Improve TestDFSIO to support different filesystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569954#comment-17569954 ] Steve Loughran commented on MAPREDUCE-7398: --- moved to the right project; reassigned. now, because i want to target hadoop 3.3.5, i have an extra feature i want here for maximum benchmark utility;j collection of IOStatistics from the run once HADOOP-17461 is in (soon!) s3a and very soon abfs will collect thread level iostats. this benchmark should be able to also report back in the reports the serialized IOStatisticsSnapshot, which would then be accumulated and reported the same way. # before each map operation: reset the IOStatisticsContext # at the end of it, snapshot to IOStatisticsSnapshot # convert to json, then to byte array and finally base 64 for embedding in the .text file on a single line (yes, this awful, but the mappers/reducers all send text records) # aggregate by coverting back tp snapshot, aggregating with the others, reserializing # collect in main process # and report with the help of IOStatisticsLogging.ioStatisticsToPrettyString Am i adding a lot of work? yes. but i know you oracle engineers want to add io stats collection and reporting, if you haven't already, as well as the hadoop abfs and s3a connectorsl, google GCS has done it in in their code... > Improve TestDFSIO to support different filesystem > - > > Key: MAPREDUCE-7398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7398 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: benchmarks >Affects Versions: 3.3.3 >Reporter: Vijayakumar Govindasamy >Assignee: Vijayakumar Govindasamy >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > TestDFSIO is the tool used for performance benchmarking on Distributed File > System. Recently there is a support added for different file system in HDFS. > Ex: s3:// (from Amazon), oci:// (from oracle). > > Request is to improve the TestDFSIO code to support newly added filesystem so > that it can be used for benchmarking IOPS and Throughput of those > filesystems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7398) Improve TestDFSIO to support different filesystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned MAPREDUCE-7398: - Assignee: Vijayakumar Govindasamy > Improve TestDFSIO to support different filesystem > - > > Key: MAPREDUCE-7398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7398 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: benchmarks >Affects Versions: 3.3.3 >Reporter: Vijayakumar Govindasamy >Assignee: Vijayakumar Govindasamy >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > TestDFSIO is the tool used for performance benchmarking on Distributed File > System. Recently there is a support added for different file system in HDFS. > Ex: s3:// (from Amazon), oci:// (from oracle). > > Request is to improve the TestDFSIO code to support newly added filesystem so > that it can be used for benchmarking IOPS and Throughput of those > filesystems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-7398) Improve TestDFSIO to support different filesystem
[ https://issues.apache.org/jira/browse/MAPREDUCE-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HDFS-16674 to MAPREDUCE-7398: -- Component/s: (was: benchmarks) (was: fs) (was: test) benchmarks Key: MAPREDUCE-7398 (was: HDFS-16674) Affects Version/s: 3.3.3 (was: 3.1.1) Project: Hadoop Map/Reduce (was: Hadoop HDFS) > Improve TestDFSIO to support different filesystem > - > > Key: MAPREDUCE-7398 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7398 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: benchmarks >Affects Versions: 3.3.3 >Reporter: Vijayakumar Govindasamy >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > TestDFSIO is the tool used for performance benchmarking on Distributed File > System. Recently there is a support added for different file system in HDFS. > Ex: s3:// (from Amazon), oci:// (from oracle). > > Request is to improve the TestDFSIO code to support newly added filesystem so > that it can be used for benchmarking IOPS and Throughput of those > filesystems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7397) Received status code 501 from server: HTTPS Required
[
https://issues.apache.org/jira/browse/MAPREDUCE-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568152#comment-17568152
]
Steve Loughran commented on MAPREDUCE-7397:
---
looks like related to this. https://github.com/jakartaee/rest/issues/571
> Received status code 501 from server: HTTPS Required
>
>
> Key: MAPREDUCE-7397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, client
>Affects Versions: 3.3.3
>Reporter: Jitin Dominic
>Priority: Major
>
> I've updated my _hadoop-client_ dependency to v3.3.3. While using
> {_}gradle{_}(v1.10) to build my jars, I keep getting following error:
> {code:java}
> org.apache.hadoop:hadoop-client:3.3.3 >
> org.apache.hadoop:hadoop-mapreduce-client-jobclient:3.3.3 >
> org.apache.hadoop:hadoop-mapreduce-client-common:3.3.3
> Could not HEAD
> 'http://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom'.
> Received status code 501 from server: HTTPS Required
> > Could not parse POM
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom
> > Illegal character in path at index 88:
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> > Could not parse POM
> https://repo.grails.org/grails/core/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom
> > Illegal character in path at index 93:
> https://repo.grails.org/grails/core/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> {code}
>
> Is there a work-around for this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7397) Received status code 501 from server: HTTPS Required
[
https://issues.apache.org/jira/browse/MAPREDUCE-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved MAPREDUCE-7397.
---
Resolution: Cannot Reproduce
> Received status code 501 from server: HTTPS Required
>
>
> Key: MAPREDUCE-7397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, client
>Affects Versions: 3.3.3
>Reporter: Jitin Dominic
>Priority: Major
>
> I've updated my _hadoop-client_ dependency to v3.3.3. While using
> {_}gradle{_}(v1.10) to build my jars, I keep getting following error:
> {code:java}
> org.apache.hadoop:hadoop-client:3.3.3 >
> org.apache.hadoop:hadoop-mapreduce-client-jobclient:3.3.3 >
> org.apache.hadoop:hadoop-mapreduce-client-common:3.3.3
> Could not HEAD
> 'http://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom'.
> Received status code 501 from server: HTTPS Required
> > Could not parse POM
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom
> > Illegal character in path at index 88:
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> > Could not parse POM
> https://repo.grails.org/grails/core/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom
> > Illegal character in path at index 93:
> https://repo.grails.org/grails/core/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> {code}
>
> Is there a work-around for this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7397) Received status code 501 from server: HTTPS Required
[
https://issues.apache.org/jira/browse/MAPREDUCE-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17567909#comment-17567909
]
Steve Loughran commented on MAPREDUCE-7397:
---
that's your maven repository playing up, nothing to do with hadoop. for some
reason your build wants to go to an http url and gradle is blocking it for
security reasons. WORKSFORME.
rs api dependency is cut for other reasons in 3.3.4, check the RC when it comes
out this week. HADOOP-18332
Till then, review/fix your list of maven repos
> Received status code 501 from server: HTTPS Required
>
>
> Key: MAPREDUCE-7397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, client
>Affects Versions: 3.3.3
>Reporter: Jitin Dominic
>Priority: Major
>
> I've updated my _hadoop-client_ dependency to v3.3.3. While using
> {_}gradle{_}(v1.10) to build my jars, I keep getting following error:
> {code:java}
> org.apache.hadoop:hadoop-client:3.3.3 >
> org.apache.hadoop:hadoop-mapreduce-client-jobclient:3.3.3 >
> org.apache.hadoop:hadoop-mapreduce-client-common:3.3.3
> Could not HEAD
> 'http://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom'.
> Received status code 501 from server: HTTPS Required
> > Could not parse POM
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom
> > Illegal character in path at index 88:
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> > Could not parse POM
> https://repo.grails.org/grails/core/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom
> > Illegal character in path at index 93:
> https://repo.grails.org/grails/core/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> {code}
>
> Is there a work-around for this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7397) Received status code 501 from server: HTTPS Required
[
https://issues.apache.org/jira/browse/MAPREDUCE-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7397:
--
Component/s: build
> Received status code 501 from server: HTTPS Required
>
>
> Key: MAPREDUCE-7397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, client
>Affects Versions: 3.3.3
>Reporter: Jitin Dominic
>Priority: Major
>
> I've updated my _hadoop-client_ dependency to v3.3.3. While using
> {_}gradle{_}(v1.10) to build my jars, I keep getting following error:
> {code:java}
> org.apache.hadoop:hadoop-client:3.3.3 >
> org.apache.hadoop:hadoop-mapreduce-client-jobclient:3.3.3 >
> org.apache.hadoop:hadoop-mapreduce-client-common:3.3.3
> Could not HEAD
> 'http://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom'.
> Received status code 501 from server: HTTPS Required
> > Could not parse POM
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom
> > Illegal character in path at index 88:
> https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> > Could not parse POM
> https://repo.grails.org/grails/core/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.pom
> > Illegal character in path at index 93:
> https://repo.grails.org/grails/core/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.${packaging.type}
> {code}
>
> Is there a work-around for this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7371) DistributedCache alternative APIs should not use DistributedCache APIs internally
[ https://issues.apache.org/jira/browse/MAPREDUCE-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7371: -- Fix Version/s: 3.3.9 > DistributedCache alternative APIs should not use DistributedCache APIs > internally > - > > Key: MAPREDUCE-7371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7371 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > DistributedCache has been deprecated long back and it's still being used as > of today. However, all the alternative Job APIs (for deprecated > DistributedCache APIs) still internally use DistributedCache APIs only, this > is a deadlock and it leaves no room for removal of DistributedCache APIs in > future. We should move core logic to Job or JobContext as required and let > deprecated DistributedCache APIs point to the right alternatives internally. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7391) TestLocalDistributedCacheManager failing after HADOOP-16202
[ https://issues.apache.org/jira/browse/MAPREDUCE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7391. --- Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed > TestLocalDistributedCacheManager failing after HADOOP-16202 > --- > > Key: MAPREDUCE-7391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7391 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0, 3.3.9 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > Time Spent: 1.5h > Remaining Estimate: 0h > > After HADOOP-16202, TestLocalDistributedCacheManager.testDownload is failing > with an NPE -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-7391) TestLocalDistributedCacheManager failing after HADOOP-16202
[ https://issues.apache.org/jira/browse/MAPREDUCE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved YARN-11189 to MAPREDUCE-7391: -- Component/s: test (was: test) Key: MAPREDUCE-7391 (was: YARN-11189) Affects Version/s: 3.4.0 (was: 3.4.0) Project: Hadoop Map/Reduce (was: Hadoop YARN) > TestLocalDistributedCacheManager failing after HADOOP-16202 > --- > > Key: MAPREDUCE-7391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7391 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > After HADOOP-16202, TestLocalDistributedCacheManager.testDownload is failing > with an NPE -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7391) TestLocalDistributedCacheManager failing after HADOOP-16202
[ https://issues.apache.org/jira/browse/MAPREDUCE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-7391: -- Affects Version/s: 3.3.4 > TestLocalDistributedCacheManager failing after HADOOP-16202 > --- > > Key: MAPREDUCE-7391 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7391 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.4.0, 3.3.4 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > After HADOOP-16202, TestLocalDistributedCacheManager.testDownload is failing > with an NPE -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7331) Make temporary directory used by FileOutputCommitter configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537508#comment-17537508
]
Steve Loughran commented on MAPREDUCE-7331:
---
update on this.
* the manifest committer is in branch-3.3 and will be in the next hadoop
release off that branch.
* it's also in the cloudera 7.2.15 runtime which shipped last week, though as
preview rather than the default in azure and gcs deployments.
The committeer should be able to coexist who with other Jobs happening in
parallel; it's currently you will have to disable job clean up for that
coexistence to work.
I'll be happy to review and merge a PR which restricts that temp dir cleanup to
_temporary/$jobID, so allow multiple I jobs to store the intermediate work
side-by-side.
however, whoever supplies a PR to do this Wwill have to provide evidence that
job commit will be safe in parallel execution, at least where the separate jobs
are all writing data into the same partition tree. I would recommend reading "a
zero rename committer" before trying to do so as we really do need a rigorous
analysis here. Distributed commit protocols are hard!
> Make temporary directory used by FileOutputCommitter configurable
> -
>
> Key: MAPREDUCE-7331
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7331
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
>Affects Versions: 3.0.0
> Environment: CDH 6.2.1 Hadoop 3.0.0
>Reporter: Bimalendu Choudhary
>Priority: Major
>
> Spark SQL applications uses FileOutputCommitter to commit and merge its files
> under a table directory. The hardcoded PENDING_DIR_NAME = _temporary
> directory results in multiple application using the same temporary directory.
> This casues unwanted results of one application interfering with other
> applications temporary files. Also one application ending up deleting
> temporary files of other. There is no way right now for applications to have
> there unique path to store the temporary files to avoid any interference from
> other totally independent applications. I think the temporary directory
> being used by FileOutputCommitter should be made configurable to let the
> caller call with with its own unique value as per the requirement and avoid
> it getting deleted or overwritten by other applications
> Something like:
> {quote}public static final String PENDING_DIR_NAME_DEFAULT = "_temporary";
> public static final String PENDING_DIR_NAME_DEFAULT =
> "mapreduce.fileoutputcommitter.tempdir";
> {quote}
>
> This can be used very efficiently by Spark applications to handle even stage
> failures where temporary directories from previous attempts cause problem and
> can help in so many situations.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7378) An error occurred while concurrently writing to a path
[
https://issues.apache.org/jira/browse/MAPREDUCE-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved MAPREDUCE-7378.
---
Resolution: Won't Fix
> An error occurred while concurrently writing to a path
> --
>
> Key: MAPREDUCE-7378
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7378
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: jingpan xiong
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When we use FileOutputCommitter as the base class of Job Committer for other
> components, we may meet an concurrently writing problem.
> Like HadoopMapReduceCommitProtocol in Spark, when there have multiple
> application to write data in same path, they will commit job and task in the
> "_temporary" dir. Once a Job finished ,it will delete the "_temporary" dir,
> make the other jobs failed.
>
> error message:
> {code:java}
> // code placeholder
> 21/11/04 19:01:21 ERROR Utils: Aborting task ExitCodeException exitCode=1:
> chmod: cannot access
> '/data/spark-examples/spark-warehouse/test/temporary/0/_temporary/attempt_202111041901182933014038999149736_0001_m_01
>
> 4/dt=2021-11-03/hour=10/.part-1-95895b03-45d2-4ac6-806b-b76fd1dfa3dc.c000.snappy.parquet.crc':
> No such file or directory at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.util.Shell.execCommand(Shell.java:1307) at
> org.apache.hadoop.util.Shell.execCommand(Shell.java:1289) at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:978)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:324)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:294)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:439)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:428)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:459)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:437)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:521)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:500)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195) at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175) at
> org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)
> at
> org.apache.parquet.hadoop.ParquetFileWriter.(ParquetFileWriter.java:329)
> at
> org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:482)
> at
> org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:420)
> at
> org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:409)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.(ParquetOutputWriter.scala:36)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:150)
> at
> org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.renewCurrentWriter(FileFormatDataWriter.scala:290)
> at
> org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:357)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:304)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:311)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$16(FileFormatWriter.scala:229)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at
> org.apache.spark.scheduler.Task.run(Task.scala:131) at
> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
[jira] [Commented] (MAPREDUCE-7378) An error occurred while concurrently writing to a path
[
https://issues.apache.org/jira/browse/MAPREDUCE-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537504#comment-17537504
]
Steve Loughran commented on MAPREDUCE-7378:
---
this is a duplicate of MAPREDUCE-7331. see discussions there.
we aren't going to accept any changes which go near FileOutputCommitter. there
is too much of breaking things, even with "minor" changes. it's a brittle and
over complex piece of code.
it was also never designed for concurrent job commits. editing the temporary
dir path hides one problem but it is not sufficient
* v2 commit is broken and unsafe anywnere where you aren't happy with the
output from a failed task attempt being included in your final output.
* v1 can't handle concurrent job commit. the v1 job commit assumes exclusive
writes and so can rename directory trees from job attempt dir to dest dur.
> An error occurred while concurrently writing to a path
> --
>
> Key: MAPREDUCE-7378
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7378
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: jingpan xiong
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When we use FileOutputCommitter as the base class of Job Committer for other
> components, we may meet an concurrently writing problem.
> Like HadoopMapReduceCommitProtocol in Spark, when there have multiple
> application to write data in same path, they will commit job and task in the
> "_temporary" dir. Once a Job finished ,it will delete the "_temporary" dir,
> make the other jobs failed.
>
> error message:
> {code:java}
> // code placeholder
> 21/11/04 19:01:21 ERROR Utils: Aborting task ExitCodeException exitCode=1:
> chmod: cannot access
> '/data/spark-examples/spark-warehouse/test/temporary/0/_temporary/attempt_202111041901182933014038999149736_0001_m_01
>
> 4/dt=2021-11-03/hour=10/.part-1-95895b03-45d2-4ac6-806b-b76fd1dfa3dc.c000.snappy.parquet.crc':
> No such file or directory at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at
> org.apache.hadoop.util.Shell.run(Shell.java:901) at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at
> org.apache.hadoop.util.Shell.execCommand(Shell.java:1307) at
> org.apache.hadoop.util.Shell.execCommand(Shell.java:1289) at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:978)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:324)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:294)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:439)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:428)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:459)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:437)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:521)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:500)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195) at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175) at
> org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)
> at
> org.apache.parquet.hadoop.ParquetFileWriter.(ParquetFileWriter.java:329)
> at
> org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:482)
> at
> org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:420)
> at
> org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:409)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.(ParquetOutputWriter.scala:36)
> at
> org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:150)
> at
> org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.renewCurrentWriter(FileFormatDataWriter.scala:290)
> at
> org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:357)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
> at
> org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:304)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496)
> at
>
[jira] [Resolved] (MAPREDUCE-7373) Building MapReduce NativeTask fails on Fedora 34+
[
https://issues.apache.org/jira/browse/MAPREDUCE-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved MAPREDUCE-7373.
---
Fix Version/s: 3.3.3
(was: 3.4.0)
(was: 3.3.4)
Resolution: Fixed
FIxed in 3.3.3; updating fix versions as appropriate
> Building MapReduce NativeTask fails on Fedora 34+
> -
>
> Key: MAPREDUCE-7373
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7373
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: build, nativetask
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.4, 3.3.3
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Fedora 34 adopted GCC 11, in which C++17 features are enabled by default.
> https://gcc.gnu.org/projects/cxx-status.html#cxx17
> Building MapReduce NativeTask with it leads to the following error.
> (I found it on branch-3.2, but it's supposed to be the same as trunk)
> {code}
> $ mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
> ...
> [WARNING] In file included from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MapOutputCollector.h:30,
> [WARNING] from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/handler/MCollectorOutputHandler.cc:24:
> [WARNING]
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/PartitionBucket.h:127:36:
> error: ISO C++17 does not allow dynamic exception specifications
> [WARNING] 127 | void spill(IFileWriter * writer) throw (IOException,
> UnsupportException);
> [WARNING] |^
> [WARNING] make[2]: *** [CMakeFiles/nativetask_static.dir/build.make:160:
> CMakeFiles/nativetask_static.dir/main/native/src/handler/MCollectorOutputHandler.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/Makefile2:115:
> CMakeFiles/nativetask_static.dir/all] Error 2
> [WARNING] make: *** [Makefile:91: all] Error 2
> ...
> [INFO] Apache Hadoop MapReduce HistoryServer Plugins .. SUCCESS [ 0.570
> s]
> [INFO] Apache Hadoop MapReduce NativeTask . FAILURE [ 11.016
> s]
> [INFO] Apache Hadoop MapReduce Uploader ... SKIPPED
> [INFO] Apache Hadoop MapReduce Examples ... SKIPPED
> [INFO] Apache Hadoop MapReduce SKIPPED
> [INFO] Apache Hadoop MapReduce Streaming .. SKIPPED
> [INFO] Apache Hadoop Distributed Copy . SKIPPED
> [INFO] Apache Hadoop Archives . SKIPPED
> [INFO] Apache Hadoop Archive Logs . SKIPPED
> [INFO] Apache Hadoop Rumen SKIPPED
> [INFO] Apache Hadoop Gridmix .. SKIPPED
> [INFO] Apache Hadoop Data Join SKIPPED
> [INFO] Apache Hadoop Extras ... SKIPPED
> [INFO] Apache Hadoop Pipes SKIPPED
> [INFO] Apache Hadoop OpenStack support SKIPPED
> [INFO] Apache Hadoop Amazon Web Services support .. SKIPPED
> [INFO] Apache Hadoop Kafka Library support SKIPPED
> [INFO] Apache Hadoop Azure support SKIPPED
> [INFO] Apache Hadoop Aliyun OSS support ... SKIPPED
> [INFO] Apache Hadoop Client Aggregator SKIPPED
> [INFO] Apache Hadoop Scheduler Load Simulator . SKIPPED
> [INFO] Apache Hadoop Resource Estimator Service ... SKIPPED
> [INFO] Apache Hadoop Azure Data Lake support .. SKIPPED
> [INFO] Apache Hadoop Tools Dist ... SKIPPED
> [INFO] Apache Hadoop Tools SKIPPED
> [INFO] Apache Hadoop Client API ... SKIPPED
> [INFO] Apache Hadoop Client Runtime ... SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants .. SKIPPED
> [INFO] Apache Hadoop Client Test Minicluster .. SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
> [INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
> [INFO] Apache Hadoop Distribution . SKIPPED
> [INFO] Apache Hadoop Client Modules ... SKIPPED
> [INFO] Apache Hadoop Cloud Storage SKIPPED
> [INFO] Apache Hadoop Cloud Storage Project SKIPPED
> [INFO]
>
[jira] [Reopened] (MAPREDUCE-7373) Building MapReduce NativeTask fails on Fedora 34+
[
https://issues.apache.org/jira/browse/MAPREDUCE-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reopened MAPREDUCE-7373:
---
> Building MapReduce NativeTask fails on Fedora 34+
> -
>
> Key: MAPREDUCE-7373
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7373
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: build, nativetask
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Fedora 34 adopted GCC 11, in which C++17 features are enabled by default.
> https://gcc.gnu.org/projects/cxx-status.html#cxx17
> Building MapReduce NativeTask with it leads to the following error.
> (I found it on branch-3.2, but it's supposed to be the same as trunk)
> {code}
> $ mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
> ...
> [WARNING] In file included from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MapOutputCollector.h:30,
> [WARNING] from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/handler/MCollectorOutputHandler.cc:24:
> [WARNING]
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/PartitionBucket.h:127:36:
> error: ISO C++17 does not allow dynamic exception specifications
> [WARNING] 127 | void spill(IFileWriter * writer) throw (IOException,
> UnsupportException);
> [WARNING] |^
> [WARNING] make[2]: *** [CMakeFiles/nativetask_static.dir/build.make:160:
> CMakeFiles/nativetask_static.dir/main/native/src/handler/MCollectorOutputHandler.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/Makefile2:115:
> CMakeFiles/nativetask_static.dir/all] Error 2
> [WARNING] make: *** [Makefile:91: all] Error 2
> ...
> [INFO] Apache Hadoop MapReduce HistoryServer Plugins .. SUCCESS [ 0.570
> s]
> [INFO] Apache Hadoop MapReduce NativeTask . FAILURE [ 11.016
> s]
> [INFO] Apache Hadoop MapReduce Uploader ... SKIPPED
> [INFO] Apache Hadoop MapReduce Examples ... SKIPPED
> [INFO] Apache Hadoop MapReduce SKIPPED
> [INFO] Apache Hadoop MapReduce Streaming .. SKIPPED
> [INFO] Apache Hadoop Distributed Copy . SKIPPED
> [INFO] Apache Hadoop Archives . SKIPPED
> [INFO] Apache Hadoop Archive Logs . SKIPPED
> [INFO] Apache Hadoop Rumen SKIPPED
> [INFO] Apache Hadoop Gridmix .. SKIPPED
> [INFO] Apache Hadoop Data Join SKIPPED
> [INFO] Apache Hadoop Extras ... SKIPPED
> [INFO] Apache Hadoop Pipes SKIPPED
> [INFO] Apache Hadoop OpenStack support SKIPPED
> [INFO] Apache Hadoop Amazon Web Services support .. SKIPPED
> [INFO] Apache Hadoop Kafka Library support SKIPPED
> [INFO] Apache Hadoop Azure support SKIPPED
> [INFO] Apache Hadoop Aliyun OSS support ... SKIPPED
> [INFO] Apache Hadoop Client Aggregator SKIPPED
> [INFO] Apache Hadoop Scheduler Load Simulator . SKIPPED
> [INFO] Apache Hadoop Resource Estimator Service ... SKIPPED
> [INFO] Apache Hadoop Azure Data Lake support .. SKIPPED
> [INFO] Apache Hadoop Tools Dist ... SKIPPED
> [INFO] Apache Hadoop Tools SKIPPED
> [INFO] Apache Hadoop Client API ... SKIPPED
> [INFO] Apache Hadoop Client Runtime ... SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants .. SKIPPED
> [INFO] Apache Hadoop Client Test Minicluster .. SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
> [INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
> [INFO] Apache Hadoop Distribution . SKIPPED
> [INFO] Apache Hadoop Client Modules ... SKIPPED
> [INFO] Apache Hadoop Cloud Storage SKIPPED
> [INFO] Apache Hadoop Cloud Storage Project SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 04:10 min
> [INFO]
[jira] [Updated] (MAPREDUCE-7373) Building MapReduce NativeTask fails on Fedora 34+
[
https://issues.apache.org/jira/browse/MAPREDUCE-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7373:
--
Target Version/s: 3.3.3
> Building MapReduce NativeTask fails on Fedora 34+
> -
>
> Key: MAPREDUCE-7373
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7373
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: build, nativetask
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Fedora 34 adopted GCC 11, in which C++17 features are enabled by default.
> https://gcc.gnu.org/projects/cxx-status.html#cxx17
> Building MapReduce NativeTask with it leads to the following error.
> (I found it on branch-3.2, but it's supposed to be the same as trunk)
> {code}
> $ mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
> ...
> [WARNING] In file included from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/MapOutputCollector.h:30,
> [WARNING] from
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/handler/MCollectorOutputHandler.cc:24:
> [WARNING]
> /home/vagrant/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib/PartitionBucket.h:127:36:
> error: ISO C++17 does not allow dynamic exception specifications
> [WARNING] 127 | void spill(IFileWriter * writer) throw (IOException,
> UnsupportException);
> [WARNING] |^
> [WARNING] make[2]: *** [CMakeFiles/nativetask_static.dir/build.make:160:
> CMakeFiles/nativetask_static.dir/main/native/src/handler/MCollectorOutputHandler.cc.o]
> Error 1
> [WARNING] make[1]: *** [CMakeFiles/Makefile2:115:
> CMakeFiles/nativetask_static.dir/all] Error 2
> [WARNING] make: *** [Makefile:91: all] Error 2
> ...
> [INFO] Apache Hadoop MapReduce HistoryServer Plugins .. SUCCESS [ 0.570
> s]
> [INFO] Apache Hadoop MapReduce NativeTask . FAILURE [ 11.016
> s]
> [INFO] Apache Hadoop MapReduce Uploader ... SKIPPED
> [INFO] Apache Hadoop MapReduce Examples ... SKIPPED
> [INFO] Apache Hadoop MapReduce SKIPPED
> [INFO] Apache Hadoop MapReduce Streaming .. SKIPPED
> [INFO] Apache Hadoop Distributed Copy . SKIPPED
> [INFO] Apache Hadoop Archives . SKIPPED
> [INFO] Apache Hadoop Archive Logs . SKIPPED
> [INFO] Apache Hadoop Rumen SKIPPED
> [INFO] Apache Hadoop Gridmix .. SKIPPED
> [INFO] Apache Hadoop Data Join SKIPPED
> [INFO] Apache Hadoop Extras ... SKIPPED
> [INFO] Apache Hadoop Pipes SKIPPED
> [INFO] Apache Hadoop OpenStack support SKIPPED
> [INFO] Apache Hadoop Amazon Web Services support .. SKIPPED
> [INFO] Apache Hadoop Kafka Library support SKIPPED
> [INFO] Apache Hadoop Azure support SKIPPED
> [INFO] Apache Hadoop Aliyun OSS support ... SKIPPED
> [INFO] Apache Hadoop Client Aggregator SKIPPED
> [INFO] Apache Hadoop Scheduler Load Simulator . SKIPPED
> [INFO] Apache Hadoop Resource Estimator Service ... SKIPPED
> [INFO] Apache Hadoop Azure Data Lake support .. SKIPPED
> [INFO] Apache Hadoop Tools Dist ... SKIPPED
> [INFO] Apache Hadoop Tools SKIPPED
> [INFO] Apache Hadoop Client API ... SKIPPED
> [INFO] Apache Hadoop Client Runtime ... SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants .. SKIPPED
> [INFO] Apache Hadoop Client Test Minicluster .. SKIPPED
> [INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
> [INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
> [INFO] Apache Hadoop Distribution . SKIPPED
> [INFO] Apache Hadoop Client Modules ... SKIPPED
> [INFO] Apache Hadoop Cloud Storage SKIPPED
> [INFO] Apache Hadoop Cloud Storage Project SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time:
[jira] [Resolved] (MAPREDUCE-7341) Add a task-manifest output committer for Azure and GCS
[ https://issues.apache.org/jira/browse/MAPREDUCE-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7341. --- Fix Version/s: 3.3.3 Resolution: Fixed > Add a task-manifest output committer for Azure and GCS > -- > > Key: MAPREDUCE-7341 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7341 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Fix For: 3.3.3 > > Time Spent: 39.5h > Remaining Estimate: 0h > > Add a task-manifest output committer for Azure and GCS > The S3A committers are very popular in Spark on S3, as they are both correct > and fast. > The classic FileOutputCommitter v1 and v2 algorithms are all that is > available for Azure ABFS and Google GCS, and they have limitations. > The v2 algorithm isn't safe in the presence of failed task attempt commits, > so we > recommend the v1 algorithm for Azure. But that is slow because it > sequentially lists > then renames files and directories, one-by-one. The latencies of list > and rename make things slow. > Google GCS lacks the atomic directory rename required for v1 correctness; > v2 can be used (which doesn't have the job commit performance limitations), > but it's not safe. > Proposed > * Add a new FileOutputFormat committer which uses an intermediate manifest to > pass the list of files created by a TA to the job committer. > * Job committer to parallelise reading these task manifests and submit all the > rename operations into a pool of worker threads. (also: mkdir, directory > deletions on cleanup) > * Use the committer plugin mechanism added for s3a to make this the default > committer for ABFS > (i.e. no need to make any changes to FileOutputCommitter) > * Add lots of IOStatistics instrumentation + logging of operations in the > JobCommit > for visibility of where delays are occurring. > * Reuse the S3A committer _SUCCESS JSON structure to publish IOStats & other > data > for testing/support. > This committer will be faster than the V1 algorithm because of the > parallelisation, and > because a manifest written by create-and-rename will be exclusive to a single > task > attempt, delivers the isolation which the v2 committer lacks. > This is not an attempt to do an iceberg/hudi/delta-lake style manifest-only > format > for describing the contents of a table; the final output is still a directory > tree > which must be scanned during query planning. > As such the format is still suboptimal for cloud storage -but at least we > will have > faster job execution during the commit phases. > > Note: this will also work on HDFS, where again, it should be faster than > the v1 committer. However the target is very much Spark with ABFS and GCS; no > plans to worry about MR as that simplifies the challenge of dealing with job > restart (i.e. you don't have to) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7369) MapReduce tasks timing out when spends more time on MultipleOutputs#close
[
https://issues.apache.org/jira/browse/MAPREDUCE-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17479333#comment-17479333
]
Steve Loughran commented on MAPREDUCE-7369:
---
# can this be a github PR?
# why does this need to be a config option? shouldnt heartbeats from tasks
always be a liveness?
> MapReduce tasks timing out when spends more time on MultipleOutputs#close
> -
>
> Key: MAPREDUCE-7369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7369
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: Prabhu Joseph
>Assignee: Ravuri Sushma sree
>Priority: Major
> Attachments: MAPREDUCE-7369.001.patch
>
>
> MapReduce tasks timing out when spends more time on MultipleOutputs#close.
> MultipleOutputs#closes takes more time when there are multiple files to be
> closed & there is a high latency in closing a stream.
> {code}
> 2021-11-01 02:45:08,312 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1634949471086_61268_m_001115_0:
> AttemptID:attempt_1634949471086_61268_m_001115_0 Timed out after 300 secs
> {code}
> MapReduce task timeout can be increased but it is tough to set the right
> timeout value. The timeout can be disabled with 0 but that might lead to
> hanging tasks not getting killed.
> The tasks are sending the ping every 3 seconds which are not honored by
> ApplicationMaster. It expects the status information which won't be send
> during MultipleOutputs#close. This jira is to add a config which considers
> the ping from task as part of Task Liveliness Check in the ApplicationMaster.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7331) Make temporary directory used by FileOutputCommitter configurable
[
https://issues.apache.org/jira/browse/MAPREDUCE-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7331:
--
Issue Type: New Feature (was: Bug)
> Make temporary directory used by FileOutputCommitter configurable
> -
>
> Key: MAPREDUCE-7331
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7331
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
>Affects Versions: 3.0.0
> Environment: CDH 6.2.1 Hadoop 3.0.0
>Reporter: Bimalendu Choudhary
>Priority: Major
>
> Spark SQL applications uses FileOutputCommitter to commit and merge its files
> under a table directory. The hardcoded PENDING_DIR_NAME = _temporary
> directory results in multiple application using the same temporary directory.
> This casues unwanted results of one application interfering with other
> applications temporary files. Also one application ending up deleting
> temporary files of other. There is no way right now for applications to have
> there unique path to store the temporary files to avoid any interference from
> other totally independent applications. I think the temporary directory
> being used by FileOutputCommitter should be made configurable to let the
> caller call with with its own unique value as per the requirement and avoid
> it getting deleted or overwritten by other applications
> Something like:
> {quote}public static final String PENDING_DIR_NAME_DEFAULT = "_temporary";
> public static final String PENDING_DIR_NAME_DEFAULT =
> "mapreduce.fileoutputcommitter.tempdir";
> {quote}
>
> This can be used very efficiently by Spark applications to handle even stage
> failures where temporary directories from previous attempts cause problem and
> can help in so many situations.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5907) Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469434#comment-17469434 ] Steve Loughran commented on MAPREDUCE-5907: --- abfs now does incremental listing, but not deep ones > Improve getSplits() performance for fs implementations that can utilize > performance gains from recursive listing > > > Key: MAPREDUCE-5907 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5907 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.4.0 >Reporter: Sumit Kumar >Assignee: Sumit Kumar >Priority: Major > Attachments: MAPREDUCE-5907-2.patch, MAPREDUCE-5907-3.patch, > MAPREDUCE-5907.patch > > > FileInputFormat (both mapreduce and mapred implementations) use recursive > listing while calculating splits. They however do this by doing listing level > by level. That means to discover files in /foo/bar means they do listing at > /foo/bar first to get the immediate children, then make the same call on all > immediate children for /foo/bar to discover their immediate children and so > on. This doesn't scale well for object store based fs implementations like s3 > and swift because every listStatus call ends up being a webservice call to > backend. In cases where large number of files are considered for input, this > makes getSplits() call slow. > This patch adds a new set of recursive list apis that gives opportunity to > the fs implementations to optimize. The behavior remains the same for other > implementations (that is a default implementation is provided for other fs so > they don't have to implement anything new). However for objectstore based fs > implementations it provides a simple change to include recursive flag as true > (as shown in the patch) to improve listing performance. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7369) MapReduce tasks timing out when spends more time on MultipleOutputs#close
[
https://issues.apache.org/jira/browse/MAPREDUCE-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448662#comment-17448662
]
Steve Loughran commented on MAPREDUCE-7369:
---
I can see this matters for cloud deployments where at that final write I will
include blocking for all pending blocks being uploaded.
Have you thought about also parallelising the close so that and the different
outputs can be closed simultaneously?
> MapReduce tasks timing out when spends more time on MultipleOutputs#close
> -
>
> Key: MAPREDUCE-7369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7369
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.3.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> MapReduce tasks timing out when spends more time on MultipleOutputs#close.
> MultipleOutputs#closes takes more time when there are multiple files to be
> closed & there is a high latency in closing a stream.
> {code}
> 2021-11-01 02:45:08,312 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1634949471086_61268_m_001115_0:
> AttemptID:attempt_1634949471086_61268_m_001115_0 Timed out after 300 secs
> {code}
> MapReduce task timeout can be increased but it is tough to set the right
> timeout value. The timeout can be disabled with 0 but that might lead to
> hanging tasks not getting killed.
> The tasks are sending the ping every 3 seconds which are not honored by
> ApplicationMaster. It expects the status information which won't be send
> during MultipleOutputs#close. This jira is to add a config which considers
> the ping from task as part of Task Liveliness Check in the ApplicationMaster.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7367) Parallelize file moves in FileOutputCommitter v1 job commit
Steve Loughran created MAPREDUCE-7367: - Summary: Parallelize file moves in FileOutputCommitter v1 job commit Key: MAPREDUCE-7367 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7367 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 3.4.0 Reporter: Steve Loughran Assignee: Rajesh Balamohan add options to v1 job commit to scan TA dirs and rename files in parallel. This is work by Rajesh Balamohan which is an interim patch before MAPREDUCE-7341 -I Don't intend to merge it). It will be the commit before HADOOP-17981 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7366) FileOutputCommitter Enable Concurrent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7366:
--
Summary: FileOutputCommitter Enable Concurrent Writes (was:
FileOutputCommitter Enable Concurent Writes )
> FileOutputCommitter Enable Concurrent Writes
> -
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.3.1
>Reporter: ismail
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7366) FileOutputCommitter Enable Concurent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved MAPREDUCE-7366.
---
Resolution: Duplicate
topic already covered in MAPREDUCE-7331
> FileOutputCommitter Enable Concurent Writes
>
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.3.1
>Reporter: ismail
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Reopened] (MAPREDUCE-7366) FileOutputCommitter Enable Concurent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reopened MAPREDUCE-7366:
---
> FileOutputCommitter Enable Concurent Writes
>
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.3.1
>Reporter: ismail
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7341) Add a task-manifest output committer for Azure and GCS
[ https://issues.apache.org/jira/browse/MAPREDUCE-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17434443#comment-17434443 ] Steve Loughran commented on MAPREDUCE-7341: --- PR contains etag collection and use in recovery from rename failures. HADOOP-17979 could go in separately > Add a task-manifest output committer for Azure and GCS > -- > > Key: MAPREDUCE-7341 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7341 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: client >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 17h 10m > Remaining Estimate: 0h > > Add a task-manifest output committer for Azure and GCS > The S3A committers are very popular in Spark on S3, as they are both correct > and fast. > The classic FileOutputCommitter v1 and v2 algorithms are all that is > available for Azure ABFS and Google GCS, and they have limitations. > The v2 algorithm isn't safe in the presence of failed task attempt commits, > so we > recommend the v1 algorithm for Azure. But that is slow because it > sequentially lists > then renames files and directories, one-by-one. The latencies of list > and rename make things slow. > Google GCS lacks the atomic directory rename required for v1 correctness; > v2 can be used (which doesn't have the job commit performance limitations), > but it's not safe. > Proposed > * Add a new FileOutputFormat committer which uses an intermediate manifest to > pass the list of files created by a TA to the job committer. > * Job committer to parallelise reading these task manifests and submit all the > rename operations into a pool of worker threads. (also: mkdir, directory > deletions on cleanup) > * Use the committer plugin mechanism added for s3a to make this the default > committer for ABFS > (i.e. no need to make any changes to FileOutputCommitter) > * Add lots of IOStatistics instrumentation + logging of operations in the > JobCommit > for visibility of where delays are occurring. > * Reuse the S3A committer _SUCCESS JSON structure to publish IOStats & other > data > for testing/support. > This committer will be faster than the V1 algorithm because of the > parallelisation, and > because a manifest written by create-and-rename will be exclusive to a single > task > attempt, delivers the isolation which the v2 committer lacks. > This is not an attempt to do an iceberg/hudi/delta-lake style manifest-only > format > for describing the contents of a table; the final output is still a directory > tree > which must be scanned during query planning. > As such the format is still suboptimal for cloud storage -but at least we > will have > faster job execution during the commit phases. > > Note: this will also work on HDFS, where again, it should be faster than > the v1 committer. However the target is very much Spark with ABFS and GCS; no > plans to worry about MR as that simplifies the challenge of dealing with job > restart (i.e. you don't have to) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7366) FileOutputCommitter Enable Concurent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7366:
--
Component/s: mrv2
> FileOutputCommitter Enable Concurent Writes
>
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.3.1
>Reporter: ismail
>Priority: Major
> Labels: committers, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7366) FileOutputCommitter Enable Concurent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved MAPREDUCE-7366.
---
Resolution: Won't Fix
closing as a wontfix as
* v2 is broken
* this doesn't work with v1 as v1 job commit assumes exclusive access to the
dest dir
* general fear of going near FileOutputCommitter
Proposed a patch in MAPREDUCE-7366
> FileOutputCommitter Enable Concurent Writes
>
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: ismail
>Priority: Major
> Labels: committers, easyfix, easytask, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7366) FileOutputCommitter Enable Concurent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7366:
--
Affects Version/s: 3.3.1
> FileOutputCommitter Enable Concurent Writes
>
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 3.3.1
>Reporter: ismail
>Priority: Major
> Labels: committers, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7366) FileOutputCommitter Enable Concurent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7366:
--
Labels: committers pull-request-available (was: committers easyfix
easytask pull-request-available)
> FileOutputCommitter Enable Concurent Writes
>
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: ismail
>Priority: Major
> Labels: committers, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7366) FileOutputCommitter Enable Concurent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7366:
--
Labels: pull-request-available (was: committers pull-request-available)
> FileOutputCommitter Enable Concurent Writes
>
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.3.1
>Reporter: ismail
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7366) FileOutputCommitter Enable Concurent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17433853#comment-17433853
]
Steve Loughran edited comment on MAPREDUCE-7366 at 10/25/21, 4:16 PM:
--
closing as a wontfix as
* v2 is broken
* this doesn't work with v1 as v1 job commit assumes exclusive access to the
dest dir
* general fear of going near FileOutputCommitter
Proposed a patch in MAPREDUCE-7341
was (Author: ste...@apache.org):
closing as a wontfix as
* v2 is broken
* this doesn't work with v1 as v1 job commit assumes exclusive access to the
dest dir
* general fear of going near FileOutputCommitter
Proposed a patch in MAPREDUCE-7366
> FileOutputCommitter Enable Concurent Writes
>
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: ismail
>Priority: Major
> Labels: committers, easyfix, easytask, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-7366) FileOutputCommitter Enable Concurent Writes
[
https://issues.apache.org/jira/browse/MAPREDUCE-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran moved HADOOP-17977 to MAPREDUCE-7366:
Key: MAPREDUCE-7366 (was: HADOOP-17977)
Project: Hadoop Map/Reduce (was: Hadoop Common)
> FileOutputCommitter Enable Concurent Writes
>
>
> Key: MAPREDUCE-7366
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7366
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: ismail
>Priority: Major
> Labels: committers, easyfix, easytask, pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> is it possible to make `{{PENDING_DIR_NAME}}` configurable?
> That will enable concurrent writes to same location. current if two spark
> processes write same destination one of them is failing.
> current
> {code:java}
> public static final String PENDING_DIR_NAME = "_temporary";{code}
> new:
> {code:java}
> PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir",
> "_temporary");{code}
> here is custom commiter doing it:
> https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7287) Distcp will delete existing file , If we use "-delete and -update" options and distcp file.
[
https://issues.apache.org/jira/browse/MAPREDUCE-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-7287:
--
Summary: Distcp will delete existing file , If we use "-delete and
-update" options and distcp file. (was: Distcp will delete exists file , If
we use "-delete and -update" options and distcp file.)
> Distcp will delete existing file , If we use "-delete and -update" options
> and distcp file.
>
>
> Key: MAPREDUCE-7287
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7287
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Attachments: MAPREDUCE-7287.001.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> hdfs://ns1/tmp/a is an existing file, hdfs://ns2/tmp/a is also an existing
> file.
> When I run this command,
> {code:java}
> hadoop distcp -delete -update hdfs://ns1/tmp/a hdfs://ns2/tmp/a
> {code}
> I Found hdfs://ns2/tmp/a is deleted unpectectedly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7300) PathOutputCommitter to add method failedTaskAttemptCommitRecoverable()
[ https://issues.apache.org/jira/browse/MAPREDUCE-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7300. --- Resolution: Won't Fix mapreduce-7341 is intended to replace v2 algorithm on object stores; it will provide safe FS commit semantics and higher job commit stats. Won't be as fast as V2 as job commit now does work, but it will be a lot faster, especially on deep directory trees > PathOutputCommitter to add method failedTaskAttemptCommitRecoverable() > -- > > Key: MAPREDUCE-7300 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7300 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Related to MAPREDUCE-7282 a variant solution > # we add a new method for committers to declare whether they can recover from > a task attempt commit failure > # default = true; v2 and (external) EMR spark committers can return false > execution engine - MR, Spark, can look at this after a task attempt fails to > commit and decide what to do > recoverable: execute/commit another task attempt > non-recoverable, one of (Configured) > * warn and continue > * abort the job > with the job abort option, users would be confident that if a failure > happened during the commit phase, they'd know about it and choose how to > recover. > I'd use a long/unusual name, so that in, say, Spark, reflection to could be > used to find and call the method & so compile against older releases -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7282) MR v2 commit algorithm should be deprecated and not the default
[ https://issues.apache.org/jira/browse/MAPREDUCE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved MAPREDUCE-7282. --- Resolution: Won't Fix > MR v2 commit algorithm should be deprecated and not the default > --- > > Key: MAPREDUCE-7282 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7282 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.3.0, 3.2.1, 3.1.3, 3.3.1 >Reporter: Steve Loughran >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The v2 MR commit algorithm moves files from the task attempt dir into the > dest dir on task commit -one by one > It is therefore not atomic > # if a task commit fails partway through and another task attempt commits > -unless exactly the same filenames are used, output of the first attempt may > be included in the final result > # if a worker partitions partway through task commit, and then continues > after another attempt has committed, it may partially overwrite the output > -even when the filenames are the same > Both MR and spark assume that task commits are atomic. Either they need to > consider that this is not the case, we add a way to probe for a committer > supporting atomic task commit, and the engines both add handling for task > commit failures (probably fail job) > Better: we remove this as the default, maybe also warn when it is being used -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[ https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372847#comment-17372847 ] Steve Loughran commented on MAPREDUCE-7351: --- 1. Submit it as a github PR 2. we use SLF4J for logging' thanks > CleanupJob during handle of SIGTERM signal > -- > > Key: MAPREDUCE-7351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Shubham Gupta >Priority: Minor > Fix For: 3.3.0 > > Attachments: MAPREDUCE-7351-001.patch > > > Currently MR CleanupJob happens when the job is either successful or fail. > But during kill, it is not handled. This leaves all the temporary folders > under the output path. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7351) CleanupJob during handle of SIGTERM signal
[
https://issues.apache.org/jira/browse/MAPREDUCE-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362352#comment-17362352
]
Steve Loughran commented on MAPREDUCE-7351:
---
do you mean calling {{OutputCommitter.cleanupJob()}}? Makes sense
> CleanupJob during handle of SIGTERM signal
> --
>
> Key: MAPREDUCE-7351
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7351
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
>
> Currently MR CleanupJob happens when the job is either successful or fail.
> But during kill, it is not handled. This leaves all the temporary folders
> under the output path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7287) Distcp will delete exists file , If we use "-delete and -update" options and distcp file.
[
https://issues.apache.org/jira/browse/MAPREDUCE-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17353549#comment-17353549
]
Steve Loughran commented on MAPREDUCE-7287:
---
it's in 3.3.2; I don't see any reason not to pull into 3.3.1 if someone is
going to test the patch
> Distcp will delete exists file , If we use "-delete and -update" options and
> distcp file.
> --
>
> Key: MAPREDUCE-7287
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7287
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Attachments: MAPREDUCE-7287.001.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> hdfs://ns1/tmp/a is an existing file, hdfs://ns2/tmp/a is also an existing
> file.
> When I run this command,
> {code:java}
> hadoop distcp -delete -update hdfs://ns1/tmp/a hdfs://ns2/tmp/a
> {code}
> I Found hdfs://ns2/tmp/a is deleted unpectectedly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
