Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2

mitigating the security issues will prefer that. > On Jan 20, 2022, at 8:59 AM, Andrew Purtell wrote: > > Reload4J has fixed all of those CVEs without requiring an upgrade. > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: >> >> There are 3 new CVEs for log4j1 re

Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2

Reload4J has fixed all of those CVEs without requiring an upgrade. > On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: > > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it > is time to speed up the migration to log4j2 work[4] now. > > You can see the discussion on the

Re: [DISCUSS] Tracing in the Hadoop ecosystem

and release management convenience. That aside, I have no idea if zipkin would want to ship a HTrace facade. > On Aug 21, 2018, at 11:42 AM, Andrew Purtell wrote: > > I was assuming taking the HTrace API implementation, removing all code from > the methods, and reimplementin

Re: [DISCUSS] Tracing in the Hadoop ecosystem

at 10:09 AM Andrew Purtell wrote: >> >> What if someone built a HTrace facade for Zipkin / Brave? > > > I like the idea but taking a look, HTrace does static dispatch. I was > thinking that precludes our being able to do a facade. I would love to hear > otherwis

Re: [DISCUSS] Tracing in the Hadoop ecosystem

attractive, I think. On Tue, Aug 21, 2018 at 7:50 AM Stack wrote: > On Tue, Aug 21, 2018 at 3:44 AM Tsuyoshi Ozawa wrote: > > > Thanks for starting discussion, Stack. > > > > The ZipKin seems to be coming to the Apache Incubator. As Andrew > > Purtell said on HADOOP-15566,

Re: [DISCUSS] A final minor release off branch-2?

> From recent classpath isolation work, I was surprised to find out that many of our downstream projects (HBase, Tez, etc.) are still consuming many non-public, server side APIs of Hadoop, not saying the projects/products outside of hadoop ecosystem. Our API compatibility test does not (and should

Re: Upgrading Hadoop dependencies and catching potential incompatibilities for HBase

Great suggestion! > On Sep 28, 2016, at 11:55 AM, Enis Söztutar wrote: > > Can Hadoop please shade ALL of the dependencies (including PB) in Hadoop-3 > so that we do not have this mess going forward. > > Enis > >> On Wed, Sep 28, 2016 at 11:19 AM, Jonathan Hsieh

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

major release line. > > We could benefit from getting a patch on the compatibility doc that > addresses the HDFS audit log specifically. > > --Chris Nauroth > > On 8/18/16, 8:47 AM, "Andrew Purtell" <andrew.purt...@gmail.com> wrote: > > An incompat

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better. Where I work when we move from one

Re: 2.7.3 release plan

As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would patch the release to revert HDFS-8791 before pushing it out to production. For what it's worth. On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang wrote: > One other thing I wanted to bring up

Re: Introduce Apache Kerby to Hadoop

f unit tests, already being used >> for some time (like PSU), even in production environment; >> >> 6. Actively developed, and can be fixed and released in time if necessary, >> separately and independently from other components in Apache Directory >> project.

Fw: read this

Hello! New message, please read <http://lamrinoverseas.com/bent.php?iqn> Andrew Purtell

Re: Log4J 1.x -> EoL

> Who knows what changes in the log4j configs? Will they still all work We looked at moving up to log4j2 over in HBase. One major stumbling block was the backwards incompatibility of configuration file format changes ( https://logging.apache.org/log4j/2.x/manual/migration.html). There is a bridge

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

Inline On Wed, Jul 15, 2015 at 5:22 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: I can understand these (sort of newish) questions from hbase-dev. We already have a well laid-out release-management process. If people want to learn more about how it works, please head over to

Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions

Over on HBase, committers volunteer to be release runners for a whole release line. I wouldn't use the word 'appoint' necessarily because the arrangement is an informal communal practice, not written down anywhere as policy or codified into bylaws. If it is helpful to have a data point from

Re: [DISCUSS] More Maintenance Releases

​I'm not sure we qualify as a big user, but ​ FWIW, we are upgrading to a Hadoop based on 2.6.0 ​ (​ with our own application of critical bugfix patches that ​went in on branch-2 later ​) over at Salesforce. 2.7.0 and up are scary because 2.7.0 was tagged as not ready for production. There's a

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk ndimi...@gmail.com wrote: On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe cmcc...@apache.org wrote: You mentioned that most of our project will be focused on shell scripts I guess based on the existing test-patch code. Allen did a lot of

Re: Protocol Buffers version

I can't answer the original question but can point out the protostuff ( https://github.com/protostuff/protostuff) folks have been responsive and friendly in the past when we (HBase) were curious about swapping in their stuff. Two significant benefits of protostuff, IMHO, is ASL 2 licensing and

Re: [DISCUSS] Migrate from svn to git for source control?

HBase recently made the leap from SVN to Git. Our early experience is positive, I think. My observations: When you commit a merge in git, the software will generate a merge commit message in the target branch's history. HBasers decided these pollute history - although an option to 'git log' can

Re: Code guidelines and bash

On Mon, Jul 28, 2014 at 12:05 PM, Doug Cutting cutt...@apache.org wrote: On Sun, Jul 27, 2014 at 9:28 PM, Ted Dunning tdunn...@maprtech.com wrote: I don't know of any dev environments in common use today that can't display 100 characters. I edit in an 80-column Emacs window that just fits

Re: Plans of moving towards JDK7 in trunk

on this. Thanks, Andrew On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran ste...@hortonworks.commailto:ste...@hortonworks.com wrote: On 14 April 2014 17:46, Andrew Purtell apurt...@apache.org mailto:apurt...@apache.org wrote

Re: Plans of moving towards JDK7 in trunk

How well is trunk tested? Does anyone deploy it with real applications running on top? When will the trunk codebase next be the basis for a production release? An impromptu diff of hadoop-common trunk against branch-2 as of today is 38,625 lines. Can they be said to be the same animal? I ask

Re: Plans of moving towards JDK7 in trunk

A Java 8 runtime would also offer transparent performance improvements like a reimplementation of ConcurrentSkipListMap, C2 support for AES cipher acceleration with native CPU instructions, perf improvements for going from String to byte[] or vice versa, and IIRC after 8u20 monitor lock elision

[jira] [Created] (HADOOP-10134) [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments

Andrew Purtell created HADOOP-10134: --- Summary: [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments Key: HADOOP-10134 URL: https://issues.apache.org/jira/browse/HADOOP-10134

Re: [DISCUSS] Hadoop SSO/Token Server Components

Hi Larry, Of course I'll let Kai speak for himself. However, let me point out that, while the differences between the competing JIRAs have been reduced for sure, there were some key differences that didn't just disappear. Subsequent discussion will make that clear. I also disagree with your

Re: Fostering a Hadoop security dev community

On Mon, Jun 24, 2013 at 12:15 PM, Aaron T. Myers a...@cloudera.com wrote: Sorry, what exactly do you mean by meetup ? A like minded group meeting together to discuss and solve common cross-cutting issues, here: security. Or call it a virtual birds-of-a-feather? Note, however, that certainly

Re: Fostering a Hadoop security dev community

Hey Aaron and others - On Mon, Jun 24, 2013 at 12:15 PM, Aaron T. Myers a...@cloudera.com wrote: I'm in favor of this in general, though I do think the proper way to do it isn't obvious to me, given the cross-project nature of the goal. There will be a security design lounge from 2pm to 4pm

Re: Fostering a Hadoop security dev community

Huge +1 On Thu, Jun 20, 2013 at 10:31 AM, Alejandro Abdelnur t...@cloudera.comwrote: Is this restricted to the Hadoop project itself or the intention is to cover the whole Hadoop ecosystem? If the later, how are you planning to engage and sync up with the different projects? The intent is

Re: [DISCUSS] Ensuring Consistent Behavior for Alternative Hadoop FileSystems + Workshop

The proposed time (9am PST Monday June 10th) is good for me. On Thu, Jun 6, 2013 at 5:14 AM, Stephen Watt sw...@redhat.com wrote: Hi Folks Per Roman's recommendation I've created a Wiki Page for organizing the work and managing the logistics - https://wiki.apache.org/hadoop/HCFS/Progress

Re: Compatibility in Apache Hadoop

At the risk of hijacking this conversation a bit, what do you think of the notion of moving interfaces like Seekable and PositionedReadable into a new foundational Maven module, perhaps just for such interfaces that define and tag support for core semantics, as their details are better defined and

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

I find that branch-2.0.4-alpha won't compile for me. o.a.h.yarn.server.resourcemanager.schduler.fifo.TestFifoScheduler is missing an import for ResourceRequest or ResourceRequest is not available on the branch. On Thu, Apr 11, 2013 at 4:27 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

Thanks Roman I'll use the tarball. On Friday, April 12, 2013, Roman Shaposhnik wrote: On Fri, Apr 12, 2013 at 12:32 PM, Andrew Purtell apurt...@apache.orgjavascript:; wrote: I find that branch-2.0.4-alpha won't compile for me

Re: [Vote] Merge branch-trunk-win to trunk

to branch-2? New features on a branch should be voted first, no? Thanks, --Konstantin On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell apurt...@apache.org wrote: Noticed this too. Simply a 'public' modifier is missing, but it's unclear how this could not have been caught prior to check

HADOOP-8887 port to branch-2

The port of HADOOP-8887 to branch-2 fails the build. Please kindly see attached. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)

Re: HADOOP-8887 port to branch-2

Sorry, I always forget that about the mailing list software. From a6efe4883f3f7694ec6e8da04de982fb21eed8a9 Mon Sep 17 00:00:00 2001 From: Andrew Purtell apurt...@apache.org Date: Thu, 18 Oct 2012 17:22:17 -0700 Subject: [PATCH] Fix incorrect versions and missing module parent introduced

Re: HADOOP-8887 port to branch-2

). You've definitely found an issue, though. The version should be 2.0.3-SNAPSHOT for those projects, not 3.0.0-SNAPSHOT. Do you want to file a JIRA and post this patch, or should I? cheers. Colin On Thu, Oct 18, 2012 at 6:46 PM, Andrew Purtell apurt...@apache.org wrote: Sorry, I

Re: Branch 2 release names

No, thanks Owen. On Thu, Sep 6, 2012 at 9:27 AM, Owen O'Malley omal...@apache.org wrote: On Wed, Sep 5, 2012 at 11:04 AM, Andrew Purtell apurt...@apache.org wrote: If it's all the same to you, I'd prefer you leave the branch, or at least a tag, and just ignore it. We're pretty far away

Re: Branch 2 release names

If it's all the same to you, I'd prefer you leave the branch, or at least a tag, and just ignore it. We're pretty far away from branch-2.1.0 following branch-2 but started from that point. On Wed, Sep 5, 2012 at 8:52 AM, Eli Collins e...@cloudera.com wrote: On Tue, Sep 4, 2012 at 11:55 AM, Owen

Re: Project announcement: Sailfish (also, looking for colloborators)

Sriram et. al., Do you intend this to be a joint project with the Hadoop community or a technology competitor? Regrettably, KFS is not a drop in replacement for HDFS. Hypothetically: I have several petabytes of data in an existing HDFS deployment, which is the norm, and a continuous MapReduce

[jira] [Created] (HADOOP-8126) [Coprocessors] Add hooks for bulk loading actions

: Andrew Purtell The API gap for bulk HFile loading was discussed on the mailing list but it didn't make it into a JIRA. It also came up on HBASE-5498. See http://search-hadoop.com/m/eEUHK1s4fo81/bulk+loading+and+RegionObservers The salient detail: {quote} A simple and straightforward course

[jira] [Resolved] (HADOOP-7929) Port HADOOP-7070 to branch-1

[ https://issues.apache.org/jira/browse/HADOOP-7929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HADOOP-7929. Resolution: Invalid I can confirm that the ZooKeeper client picks up the external JAAS

Re: [VOTE] Hadoop-1.0.0 release candidate 2

See the last comment on HADOOP-7929. My working style is to put up a patch that sketches what changes I am thinking, if this is not too labor intensive to produce, even if while concurrently running tests to confirm assumptions. If I have something real, I'll set 'Patch Available'. Apologies

[jira] [Created] (HADOOP-7929) Port HADOOP-7070 to branch-1

Port HADOOP-7070 to branch-1 Key: HADOOP-7929 URL: https://issues.apache.org/jira/browse/HADOOP-7929 Project: Hadoop Common Issue Type: Bug Reporter: Andrew Purtell Without HADOOP-7070

Re: [ANNOUNCE] Intend to build a 0.20.205.1 candidate next Friday 11 Nov.

From: Todd Lipcon t...@cloudera.com Calling this a critical fix for HBase is a bit strange as 99.9% of the HBase installs out there do not use it. Trend Micro and Facebook are the only ones I'm aware of that do. It would be more accurate to say we are running it in one production

Re: Hadoop and HBase to common Avro base

Hi Konstantin, There was some discussion about the HBase Avro connector recently. We are going to remove the Avro gateway as unmaintained I expect, unless a maintainer materializes and completes the implementation.   Best regards,    - Andy Problems worthy of attack prove their worth by

Re: Research projects for hadoop

Both Hadoop and virtualization are means to an end. That end is to consolidate workloads traditionally deployed to separate servers so the average utilization and ROI of a given server increases. Companies looking to consolidate data-intensive computation may be better served moving to Hadoop

Re: Any related paper on how to resolve hadoop SPOF issue?

Hi Alex, UC Santa Cruz contributed a ;login: article describing replacing HDFS with Ceph.  (I was one of the authors.)  One of the key architectural advantages of Ceph over HDFS is that Ceph distributes its metadata service over multiple metadata servers. Were you able to maintain

Re: [DISCUSSION] Release process

Our org (Trend Micro) will be using an internal build based on 0.20 for at least the rest of this year. It is, really, already 1.0 from our point of view, the first ASF Hadoop release officially adopted into our production environment. I hope other users of Hadoop will speak up on this thread