[jira] [Resolved] (HADOOP-18870) CURATOR-599 change broke functionality introduced in HADOOP-18139 and HADOOP-18709
[ https://issues.apache.org/jira/browse/HADOOP-18870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth resolved HADOOP-18870. - Fix Version/s: 3.4.0 Resolution: Fixed > CURATOR-599 change broke functionality introduced in HADOOP-18139 and > HADOOP-18709 > -- > > Key: HADOOP-18870 > URL: https://issues.apache.org/jira/browse/HADOOP-18870 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.4.0, 3.3.5 >Reporter: Ferenc Erdelyi >Assignee: Ferenc Erdelyi >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > [Curator PR#391 > |https://github.com/apache/curator/pull/391/files#diff-687a4ed1252bfb4f56b3aeeb28bee4413b7df9bec4b969b72215587158ac875dR59] > introduced a default method in the ZooKeeperFactory interface, hence the > override of the 4-parameter NewZookeeper method in the HadoopZookeeperFactory > class is not taking effect due to this. > Proposing routing the 4-parameter method to a 5-parameter method, which > instantiates the ZKConfiguration as the 5th parameter. This is a non-breaking > change, as the ZKConfiguration is currently instantiated within the method. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17791) TestActivitiesManager is flaky
Szilard Nemeth created HADOOP-17791: --- Summary: TestActivitiesManager is flaky Key: HADOOP-17791 URL: https://issues.apache.org/jira/browse/HADOOP-17791 Project: Hadoop Common Issue Type: Bug Reporter: Szilard Nemeth I noticed in our internal testing environment that org.apache.hadoop.yarn.server.resourcemanager.scheduler.activities.TestActivitiesManager.testAppActivitiesTTL failed a couple of times, quite randomly. By checking the Jira and searching for the name of the class, there are some results from this year as well: [https://issues.apache.org/jira/issues/?jql=text%20~%20TestActivitiesManager%20ORDER%20BY%20updated%20DESC] I don't know exactly how to reproduce this though. Tries to run the whole test class 60 times and it hasn't failed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [E] Re: [DISCUSS] Change project style guidelines to allow line length 100
Thanks for this initiative, Sean. +1 for increasing the length to 100 characters. I can't see a VOTE thread regarding this subject. Am I missing something? Best, Szilard On Mon, May 24, 2021 at 11:49 PM Jonathan Eagles wrote: > In apache tez, formal line length is 120 characters. So, I recommend 120+ > > On Mon, May 24, 2021 at 4:46 PM Kihwal Lee .invalid> > wrote: > > > +1 for the 100 char limit. > > But I would have liked 132 columns more. :) > > > > Kihwal > > > > On Mon, May 24, 2021 at 1:46 PM Sean Busbey > > wrote: > > > > > Hi folks! > > > > > > The consensus seems pretty strongly in favor of increasing the line > > length > > > limit. Do folks still want to see a formal VOTE thread? > > > > > > > > > > On May 19, 2021, at 4:22 PM, Sean Busbey > > > wrote: > > > > > > > > Hello! > > > > > > > > What do folks think about changing our line length guidelines to > allow > > > for 100 character width? > > > > > > > > Currently, we tell folks to follow the sun style guide with some > > > exception unrelated to line length. That guide says width of 80 is the > > > standard and our current check style rules act as enforcement. > > > > > > > > Looking at the current trunk codebase our nightly build shows a total > > of > > > ~15k line length violations; it’s about 18% of identified checkstyle > > issues. > > > > > > > > The vast majority of those line length violations are <= 100 > characters > > > long. 100 characters happens to be the length for the Google Java Style > > > Guide, another commonly adopted style guide for java projects, so I > > suspect > > > these longer lines leaking past the checkstyle precommit warning might > > be a > > > reflection of committers working across multiple java codebases. > > > > > > > > I don’t feel strongly about lines being longer, but I would like to > > move > > > towards more consistent style enforcement as a project. Updating our > > > project guidance to allow for 100 character lines would reduce the > > > likelihood that folks bringing in new contributions need a precommit > test > > > cycle to get the formatting correct. > > > > > > > > Does anyone feel strongly about keeping the line length limit at 80 > > > characters? > > > > > > > > Does anyone feel strongly about contributions coming in that clear up > > > line length violations? > > > > > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > >
Re: [VOTE] Release Apache Hadoop 3.1.4 (RC4)
+1 (binding). **TEST STEPS** 1. Build from sources (see Maven / Java and OS details below) 2. Distribute Hadoop to all nodes 3. Start HDFS services + YARN services on nodes 4. Run Mapreduce pi job (QuasiMontecarlo) 5. Verifed that application was successful through YARN RM Web UI 6. Verified version of Hadoop release from YARN RM Web UI **OS version** $ cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/; BUG_REPORT_URL="https://bugs.centos.org/; CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" **Maven version** $ mvn -v Apache Maven 3.0.5 (Red Hat 3.0.5-17) Maven home: /usr/share/maven **Java version** Java version: 1.8.0_191, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64/jre Default locale: en_US, platform encoding: ANSI_X3.4-1968 OS name: "linux", version: "3.10.0-1062.el7.x86_64", arch: "amd64", family: "unix" **Maven command to build from sources** mvn clean package -Pdist -DskipTests -Dmaven.javadoc.skip=true **OTHER NOTES** 1. Had to manually install maven in order to manually compile Hadoop based on these steps: https://gist.github.com/miroslavtamas/cdca97f2eafdd6c28b844434eaa3b631 2. Had to manually install protoc + other required libraries with the following commands (in this particular order): sudo yum install -y protobuf-devel sudo yum install -y gcc gcc-c++ make sudo yum install -y openssl-devel sudo yum install -y libgsasl Thanks, Szilard On Thu, Jul 23, 2020 at 4:05 PM Masatake Iwasaki < iwasak...@oss.nttdata.co.jp> wrote: > +1 (binding). > > * verified the checksum and signature of the source tarball. > * built from source tarball with native profile on CentOS 7 and OpenJDK 8. > * built documentation and skimmed the contents. > * ran example jobs on 3 nodes docker cluster with NN-HA and RM-HA enblaed. > * launched pseudo-distributed cluster with Kerberos and SSL enabled, ran > basic EZ operation, ran example MR jobs. > * followed the reproduction step reported in HDFS-15313 to see if the > fix works. > > Thanks, > Masatake Iwasaki > > On 2020/07/21 21:50, Gabor Bota wrote: > > Hi folks, > > > > I have put together a release candidate (RC4) for Hadoop 3.1.4. > > > > * > > The RC includes in addition to the previous ones: > > * fix for HDFS-15313. Ensure inodes in active filesystem are not > > deleted during snapshot delete > > * fix for YARN-10347. Fix double locking in > > CapacityScheduler#reinitialize in branch-3.1 > > (https://issues.apache.org/jira/browse/YARN-10347) > > * the revert of HDFS-14941, as it caused > > HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. > > (https://issues.apache.org/jira/browse/HDFS-15421) > > * HDFS-15323, as requested. > > (https://issues.apache.org/jira/browse/HDFS-15323) > > * > > > > The RC is available at: > http://people.apache.org/~gabota/hadoop-3.1.4-RC4/ > > The RC tag in git is here: > > https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC4 > > The maven artifacts are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1275/ > > > > You can find my public key at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > and http://keys.gnupg.net/pks/lookup?op=get=0xB86249D83539B38C > > > > Please try the release and vote. The vote will run for 8 weekdays, > > until July 31. 2020. 23:00 CET. > > > > > > Thanks, > > Gabor > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > >
Hadoop-trunk commit jenkins job fails due to wrong version of protoc
Hi, All hadoop-trunk builds are failing because of wrong version of protoc is being used. The version we are using is 3.0.0, however we should use 2.5.0 according to the build configuration. This makes hadoop-common fail to build. Here's a failed job as an example: https://builds.apache.org/job/Hadoop-trunk-Commit/17020/console , and this is the error message from Maven: [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 22.245 s (Wall Clock) [INFO] Finished at: 2019-08-01T11:12:41Z [INFO] [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 3.0.0', expected version is '2.5.0' -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-common Could someone please help me to indentify what is exactly wrong and where should we fix this issue? Thanks a lot, Szilard
Re: Any thoughts making Submarine a separate Apache project?
+1, this is a very great idea. As Hadoop repository has already grown huge and contains many projects, I think in general it's a good idea to separate projects in the early phase. On Wed, Jul 17, 2019, 08:50 runlin zhang wrote: > +1 ，That will be great ！ > > > 在 2019年7月10日，下午3:34，Xun Liu 写道： > > > > Hi all, > > > > This is Xun Liu contributing to the Submarine project for deep learning > > workloads running with big data workloads together on Hadoop clusters. > > > > There are a bunch of integrations of Submarine to other projects are > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The next > step > > of Submarine is going to integrate with more projects like Apache Arrow, > > Redis, MLflow, etc. & be able to handle end-to-end machine learning use > > cases like model serving, notebook management, advanced training > > optimizations (like auto parameter tuning, memory cache optimizations for > > large datasets for training, etc.), and make it run on other platforms > like > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY > project > > to Apache so we can put Submarine and TonY together to the same codebase > > (Page #30. > > > https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30 > > ). > > > > This expands the scope of the original Submarine project in exciting new > > ways. Toward that end, would it make sense to create a separate Submarine > > project at Apache? This can make faster adoption of Submarine, and allow > > Submarine to grow to a full-blown machine learning platform. > > > > There will be lots of technical details to work out, but any initial > > thoughts on this? > > > > Best Regards, > > Xun Liu > > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >
Re: new committer: Gabor Bota
Congrats, Gabor! On Tue, Jul 2, 2019, 01:36 Sean Mackrory wrote: > The Project Management Committee (PMC) for Apache Hadoop > has invited Gabor Bota to become a committer and we are pleased > to announce that he has accepted. > > Gabor has been working on the S3A file-system, especially on > the robustness and completeness of S3Guard to help deal with > inconsistency in object storage. I'm excited to see his work > with the community continue! > > Being a committer enables easier contribution to the > project since there is no need to go via the patch > submission process. This should enable better productivity. >
[jira] [Created] (HADOOP-15717) TGT renewal thread does not log IOException
Szilard Nemeth created HADOOP-15717: --- Summary: TGT renewal thread does not log IOException Key: HADOOP-15717 URL: https://issues.apache.org/jira/browse/HADOOP-15717 Project: Hadoop Common Issue Type: Improvement Reporter: Szilard Nemeth Assignee: Szilard Nemeth See here: https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L918 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15708) Reading values from Configuration before adding deprecations make it impossible to read value with deprecated key
Szilard Nemeth created HADOOP-15708:
---
Summary: Reading values from Configuration before adding
deprecations make it impossible to read value with deprecated key
Key: HADOOP-15708
URL: https://issues.apache.org/jira/browse/HADOOP-15708
Project: Hadoop Common
Issue Type: Bug
Reporter: Szilard Nemeth
Assignee: Szilard Nemeth
Hadoop Common contains a widely used Configuration class.
This class can handle deprecations of properties, e.g. if property 'A' gets
deprecated with an alternative property key 'B', users can access property
values with keys 'A' and 'B'.
Unfortunately, this does not work in one case.
When a config file is specified (for instance, XML) and a property is read with
the config.get() method, the config is loaded from the file at this time.
If the deprecation mapping is not yet specified by the time any config value is
retrieved and the XML config refers to a deprecated key, then the deprecation
mapping specified, the config value cannot be retrieved neither with the
deprecated nor with the new key.
The attached patch contains a testcase that reproduces this wrong behavior.
Here are the steps outlined what the testcase does:
1. Creates an XML config file with a deprecated property
2. Adds the config to the Configuration object
3. Retrieves the config with its deprecated key (it does not really matter
which property the user gets, could be any)
4. Specifies the deprecation rules including the one defined in the config
5. Prints and asserts the property retrieved from the config with both the
deprecated and the new property keys.
For reference, here is the log of one execution that actually shows what the
issue is:
{noformat}
Loaded items: 1
Looked up property value with name hadoop.zk.address: null
Looked up property value with name yarn.resourcemanager.zk-address:
dummyZkAddress
Contents of config file: [, ,
yarn.resourcemanager.zk-addressdummyZkAddress,
]
Looked up property value with name hadoop.zk.address: null
2018-08-31 10:10:06,484 INFO Configuration.deprecation
(Configuration.java:logDeprecation(1397)) - yarn.resourcemanager.zk-address is
deprecated. Instead, use hadoop.zk.address
Looked up property value with name hadoop.zk.address: null
Looked up property value with name hadoop.zk.address: null
java.lang.AssertionError:
Expected :dummyZkAddress
Actual :null
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15586) Fix wrong log statements in AbstractService
Szilard Nemeth created HADOOP-15586:
---
Summary: Fix wrong log statements in AbstractService
Key: HADOOP-15586
URL: https://issues.apache.org/jira/browse/HADOOP-15586
Project: Hadoop Common
Issue Type: Improvement
Reporter: Szilard Nemeth
Assignee: Szilard Nemeth
There are some wrong logging statements in AbstractService, here is one
example:
{code:java}
LOG.debug("noteFailure {}" + exception);
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
