Re: [MERGE] Use libcap instead of direct linux capability syscalls

[Henrik Nordstrom]

fre 2009-10-16 klockan 11:03 +1300 skrev Amos Jeffries:

> /* NP: keep these two if-endif separate. Non-Linux work perfectly well 

Sorry.. thought I had fixed that already..

> +#define PUSH_CAP(cap) cap_list[ncaps++] = (cap)
> 
> I can just see that converting to: 
> CAP_NET_ADMIN_ist[nCAP_NET_ADMINs++]=(CAP_NET_ADMIN) ...

Nope.. preprocessor is tokens based. But as this macro is farily simple
now it can just as well be expanded. I think the plan was to eventually
C++ encapsulate these details, but that's overkill here.

Updated patch attaced.

Regards
Henrik
# Bazaar merge directive format 2 (Bazaar 0.90)
# revision_id: hen...@henriknordstrom.net-20091015235726-\
#   tjj24dnri2arionc
# target_branch: http://www.squid-cache.org/bzr/squid3/trunk/
# testament_sha1: e0544b31cc7e7f4f877a1b5939e6cfe26d60bc6f
# timestamp: 2009-10-16 01:58:06 +0200
# base_revision_id: squ...@treenet.co.nz-20091015121532-\
#   hhwys6416uxebd9y
# 
# Begin patch
=== modified file 'configure.in'
--- configure.in	2009-10-15 10:12:38 +
+++ configure.in	2009-10-15 14:28:22 +
@@ -2763,7 +2763,7 @@
   fi
 ],[AC_MSG_RESULT(yes)])
 if test "x$use_caps" = "xyes"; then
-  dnl Check for libcap1 breakage or libcap2 fixed (assume broken unless found working)
+  dnl Check for libcap1 header breakage or libcap2 fixed (assume broken unless found working)
   libcap_broken=1
   AC_CHECK_HEADERS(sys/capability.h)
   AC_CACHE_CHECK([for operational libcap2], $libcap_broken,
@@ -2773,6 +2773,7 @@
]])],[libcap_broken=0],[])
   )
   AC_DEFINE_UNQUOTED([LIBCAP_BROKEN],$libcap_broken,[if libcap2 is available and not clashing with libc])
+  AC_CHECK_LIB(cap, cap_get_proc)
 fi
 
 AC_CHECK_TYPE(mtyp_t,AC_DEFINE(HAVE_MTYP_T,1,[mtyp_t is defined by the system headers]),,[#include 

=== modified file 'src/tools.cc'
--- src/tools.cc	2009-08-28 01:44:26 +
+++ src/tools.cc	2009-10-15 23:57:26 +
@@ -1241,50 +1241,40 @@
 {
 /* NP: keep these two if-endif separate. Non-Linux work perfectly well without Linux syscap support. */
 #if defined(_SQUID_LINUX_)
-
 #if HAVE_SYS_CAPABILITY_H
-#ifndef _LINUX_CAPABILITY_VERSION_1
-#define _LINUX_CAPABILITY_VERSION_1 _LINUX_CAPABILITY_VERSION
-#endif
-cap_user_header_t head = (cap_user_header_t) xcalloc(1, sizeof(*head));
-cap_user_data_t cap = (cap_user_data_t) xcalloc(1, sizeof(*cap));
-
-head->version = _LINUX_CAPABILITY_VERSION_1;
-
-if (capget(head, cap) != 0) {
-debugs(50, DBG_IMPORTANT, "Can't get current capabilities");
-} else if (head->version != _LINUX_CAPABILITY_VERSION_1) {
-debugs(50, DBG_IMPORTANT, "Invalid capability version " << head->version << " (expected " << _LINUX_CAPABILITY_VERSION_1 << ")");
+cap_t caps;
+if (keep)
+	caps = cap_get_proc();
+else
+	caps = cap_init();
+if (!caps) {
+	IpInterceptor.StopTransparency("Can't get current capabilities");
 } else {
-
-head->pid = 0;
-
-cap->inheritable = 0;
-cap->effective = (1 << CAP_NET_BIND_SERVICE);
-
-if (IpInterceptor.TransparentActive()) {
-cap->effective |= (1 << CAP_NET_ADMIN);
+	int ncaps = 0;
+	int rc = 0;
+	cap_value_t cap_list[10];
+	cap_list[ncaps++] = CAP_NET_BIND_SERVICE;
+
+	if (IpInterceptor.TransparentActive()) {
+	cap_list[ncaps++] = CAP_NET_ADMIN;
 #if LINUX_TPROXY2
-cap->effective |= (1 << CAP_NET_BROADCAST);
+	cap_list[ncaps++] = CAP_NET_BROADCAST;
 #endif
-}
-
-if (!keep)
-cap->permitted &= cap->effective;
-
-if (capset(head, cap) != 0) {
+	}
+
+	cap_clear_flag(caps, CAP_EFFECTIVE);
+	rc |= cap_set_flag(caps, CAP_EFFECTIVE, ncaps, cap_list, CAP_SET);
+	rc |= cap_set_flag(caps, CAP_PERMITTED, ncaps, cap_list, CAP_SET);
+
+if (rc || cap_set_proc(caps) != 0) {
 IpInterceptor.StopTransparency("Error enabling needed capabilities.");
 }
+	cap_free(caps);
 }
-
-xfree(head);
-xfree(cap);
-
 #else
 IpInterceptor.StopTransparency("Missing needed capability support.");
 #endif /* HAVE_SYS_CAPABILITY_H */
-
-#endif /* !defined(_SQUID_LINUX_) */
+#endif /* _SQUID_LINUX_ */
 }
 
 void *

# Begin bundle
IyBCYXphYXIgcmV2aXNpb24gYnVuZGxlIHY0CiMKQlpoOTFBWSZTWeXFlV4ABjxfgEQwef///39v
/2q+YAnPj1q+owJFAC6wnTU1lAAoAMkTQSeRPUzQnpPakeiaaPU00GgAaaBo0aAA4yZNGgNG
mIyNDEMCaNMQYjQYQAGHGTJo0Bo0xGRoYhgTRpiDEaDCAAwYlSPFBiNMnkmAhpiYgDRoMAmjJkMJ
gikRojRpqniPSaaTKbFMyMQ1PTJqNEeiD1NBtQP1BJIBNNNAhCek1MaNJpH6oGRmoyMTR6mT1Hpp
PCjhAcBA/KxRq1hVPG4pZm3lBsotwW7hoeBiSZwQhH0/L5PH/kbyD/uLuplZbxN2y1f8fZogkkyC
aYHcL90IFWcSSwYKsiE+LyjBhNwLMnJqM1M9XJ8pp0hqwfSU6dKQuYVSv7m9a9R6u/J5/hu0DwLV
RcnDIAkUnMqGQkxuEOu/zzGEwQUAhFxmfXGk80CefYwUo9wwYQBxOcXyfCzqxcymxgpzi5iJF2EY
0U55YbSigl3BGUOeSjOVXxhd+FvrRBZasfyxaKQCJ8BskUW8K+C2l+leb4cJr4W3P2YqIV4EazdE
8uc3g5hmBhATdqv1zKZMU1u7PYqz7dBuGAPCEUqgRrsCOo6uAXo6SLqriqnF9gpXSmR5HTRV9R1w
70c8uFh1Rvee+vka9RAroY78lh1wpyiWupjVyVseZEn8VOsFimsDmAQ1hiiWNDJGbAgPJC0F+0TW
++nIjzgwJ/15qUHGK6ew0DchuHpVdSHu9mKHqu83NkJSlKUv4q17+o3uk1Rct

Re: [MERGE] Use libcap instead of direct linux capability syscalls

[Amos Jeffries]

Henrik Nordstrom wrote:

The kernel interface, while some aspects of it is much simpler is also
not really meant to be called directly by applications.

The attached patch approximates the same functionality using libcap.
Differs slightly in how it sets the permitted capabilities to be kept on
uid change (explicit instead of masked), but end result is the same as
setting the capabilities won't work if these were not allowed.



/* NP: keep these two if-endif separate. Non-Linux work perfectly well 
without Linux syscap support. */

-#if defined(_SQUID_LINUX_)
-
-#if HAVE_SYS_CAPABILITY_H

The above was done so that interception does not get disabled on FreeBSD 
which now has some TPROXY support.



+#define PUSH_CAP(cap) cap_list[ncaps++] = (cap)

I can just see that converting to: 
CAP_NET_ADMIN_ist[nCAP_NET_ADMINs++]=(CAP_NET_ADMIN) ...


Otherwise good.

Amos
--
Please be using
  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19
  Current Beta Squid 3.1.0.14

Hudson build is back to normal: 3.HEAD-i386-Debian-sid #88

[noc]

See 

[MERGE] Use libcap instead of direct linux capability syscalls

[Henrik Nordstrom]

The kernel interface, while some aspects of it is much simpler is also
not really meant to be called directly by applications.

The attached patch approximates the same functionality using libcap.
Differs slightly in how it sets the permitted capabilities to be kept on
uid change (explicit instead of masked), but end result is the same as
setting the capabilities won't work if these were not allowed.




# Bazaar merge directive format 2 (Bazaar 0.90)
# revision_id: hen...@henriknordstrom.net-20091015142822-\
#   is615u5fl72d5vt3
# target_branch: http://www.squid-cache.org/bzr/squid3/trunk/
# testament_sha1: 7003f761ebaefca2b4e2fd090f186cfb0ec0357e
# timestamp: 2009-10-15 20:21:24 +0200
# base_revision_id: squ...@treenet.co.nz-20091015121532-\
#   hhwys6416uxebd9y
# 
# Begin patch
=== modified file 'configure.in'
--- configure.in	2009-10-15 10:12:38 +
+++ configure.in	2009-10-15 14:28:22 +
@@ -2763,7 +2763,7 @@
   fi
 ],[AC_MSG_RESULT(yes)])
 if test "x$use_caps" = "xyes"; then
-  dnl Check for libcap1 breakage or libcap2 fixed (assume broken unless found working)
+  dnl Check for libcap1 header breakage or libcap2 fixed (assume broken unless found working)
   libcap_broken=1
   AC_CHECK_HEADERS(sys/capability.h)
   AC_CACHE_CHECK([for operational libcap2], $libcap_broken,
@@ -2773,6 +2773,7 @@
]])],[libcap_broken=0],[])
   )
   AC_DEFINE_UNQUOTED([LIBCAP_BROKEN],$libcap_broken,[if libcap2 is available and not clashing with libc])
+  AC_CHECK_LIB(cap, cap_get_proc)
 fi
 
 AC_CHECK_TYPE(mtyp_t,AC_DEFINE(HAVE_MTYP_T,1,[mtyp_t is defined by the system headers]),,[#include 

=== modified file 'src/tools.cc'
--- src/tools.cc	2009-08-28 01:44:26 +
+++ src/tools.cc	2009-10-15 14:24:33 +
@@ -1240,51 +1240,41 @@
 restoreCapabilities(int keep)
 {
 /* NP: keep these two if-endif separate. Non-Linux work perfectly well without Linux syscap support. */
-#if defined(_SQUID_LINUX_)
-
-#if HAVE_SYS_CAPABILITY_H
-#ifndef _LINUX_CAPABILITY_VERSION_1
-#define _LINUX_CAPABILITY_VERSION_1 _LINUX_CAPABILITY_VERSION
-#endif
-cap_user_header_t head = (cap_user_header_t) xcalloc(1, sizeof(*head));
-cap_user_data_t cap = (cap_user_data_t) xcalloc(1, sizeof(*cap));
-
-head->version = _LINUX_CAPABILITY_VERSION_1;
-
-if (capget(head, cap) != 0) {
-debugs(50, DBG_IMPORTANT, "Can't get current capabilities");
-} else if (head->version != _LINUX_CAPABILITY_VERSION_1) {
-debugs(50, DBG_IMPORTANT, "Invalid capability version " << head->version << " (expected " << _LINUX_CAPABILITY_VERSION_1 << ")");
+#if defined(_SQUID_LINUX_) && HAVE_SYS_CAPABILITY_H
+cap_t caps;
+if (keep)
+	caps = cap_get_proc();
+else
+	caps = cap_init();
+if (!caps) {
+	IpInterceptor.StopTransparency("Can't get current capabilities");
 } else {
-
-head->pid = 0;
-
-cap->inheritable = 0;
-cap->effective = (1 << CAP_NET_BIND_SERVICE);
-
-if (IpInterceptor.TransparentActive()) {
-cap->effective |= (1 << CAP_NET_ADMIN);
+#define PUSH_CAP(cap) cap_list[ncaps++] = (cap)
+	int ncaps = 0;
+	int rc = 0;
+	cap_value_t cap_list[10];
+	PUSH_CAP(CAP_NET_BIND_SERVICE);
+
+	if (IpInterceptor.TransparentActive()) {
+	PUSH_CAP(CAP_NET_ADMIN);
 #if LINUX_TPROXY2
-cap->effective |= (1 << CAP_NET_BROADCAST);
+	PUSH_CAP(CAP_NET_BROADCAST);
 #endif
-}
-
-if (!keep)
-cap->permitted &= cap->effective;
-
-if (capset(head, cap) != 0) {
+	}
+#undef PUSH_CAP
+
+	cap_clear_flag(caps, CAP_EFFECTIVE);
+	rc |= cap_set_flag(caps, CAP_EFFECTIVE, ncaps, cap_list, CAP_SET);
+	rc |= cap_set_flag(caps, CAP_PERMITTED, ncaps, cap_list, CAP_SET);
+
+if (rc || cap_set_proc(caps) != 0) {
 IpInterceptor.StopTransparency("Error enabling needed capabilities.");
 }
+	cap_free(caps);
 }
-
-xfree(head);
-xfree(cap);
-
 #else
 IpInterceptor.StopTransparency("Missing needed capability support.");
 #endif /* HAVE_SYS_CAPABILITY_H */
-
-#endif /* !defined(_SQUID_LINUX_) */
 }
 
 void *

# Begin bundle
IyBCYXphYXIgcmV2aXNpb24gYnVuZGxlIHY0CiMKQlpoOTFBWSZTWduZKS4ABHffgEQwee///39v
/2q+YAf98llXqgAGd3O2GiQAAwkkChNR7U9GqM09JtMqPU9qj1PRpGI9Rk0ANB6mgBzCaA0B
o0YRoMRpiZMTQYRoGQDJgJKQDTRpTTamgAAGjQAAaAADRkAxTRCj0nqA3qQbU0GgAGnqPUAH
MJoDQGjRhGgxGmJkxNBhGgZAMmAkkBGgBAmTSNNqU20aKPU9NpqnpNHiZE0A9TbKjyXkn8xVSzOo
6nuY29xwpyJqc4QM6dfBXF6U3PALME7+yiygZHhYJ6GFAZOydW+/qhJBCOT4sE63ILSZnlhQfIiO
rm+MtaEbipK8/U+Z8+ujzoF8y4g4ciAkUoXkPMgYOJLLXM4IMFZZKMXCy+LORYBB7SuZyL0Il75E
ICDp4WDDvnrOPrEValQtaNatSpCNt54PN41VtS2WHA7zHj/2nM+Gkz7WaFCCr0K5SjhJycwMMIzT
3KF3Vi15a6qJ7dBlLhLczYCmVbNeLJxzyFNSShC+ZTDESO+q+5wDEzUZMUdKjOw7ZGXYorvDob1G
2PF+rT+n6s3T9HSiN7mWhRxV8L9gMeCCYg5JZL1dAKxAyX6LI0Qj4SZbiDgFaCOVBkFnvYa7g5zu
oRHDPzKtD2BeZhPfkPMjzyJJ6EnhowkWFetKqWUc7YI2ltKCIjGJri+NJI4BghToa8ZMPJzefxrJ
kanJVYOoyZhzJmDouttZAmOv6yoYJrJJ0ZpjEKGmVr2VjZhqjSbfBltSKFEI4QSQCncJE1PUZivx
HOGZXnwlgeOh8EnYI36GsoMXrDLTCDtk9FMV9C9IWJaPImv5VvSsW

Hudson build is back to normal: 3.HEAD-amd64-CentOS-5.3 #152

[noc]

See 

Happy Diwali to all : squid-smp

[Sachin Malave]

Hello,

Here in India we are celebrating DIWALI festival.
So I am on vacation for next 5 days. Will continue squid-smp
discussion soon

Happy Diwali to all..

-- 
Mr. S. H. Malave
Computer Science & Engineering Department,
Walchand College of Engineering,Sangli.
sachinmal...@wce.org.in

Build failed in Hudson: 3.HEAD-amd64-CentOS-5.3 #151

[noc]

See 

Changes:

[Amos Jeffries ] SourceLayout: pull ARP operations into 
libeui

This separates the current code for ARP lookup and encode/decode.
Also makes the code buildable without fatal build errors on
non-supporting systems. A loud run-time error is generated if related
features are attempt use on non-supporting systems.

FUTURE WORK:
* unit-tests
* on-demand lookups and logging of MAC/EUI-48 for a client
similar to how ident is done.
* on-demand passing MAC/EUI-48 to external_acl_type helpers
* EUI-64 for IPv6 support.
* Increased ARP support on other OS.

[Henrik Nordstrom ] Bug #2628: Correct default log 
location to ../var/logs/squid.pid (same as logs)

--
[...truncated 19739 lines...]
checking for closedir... yes
checking for opendir... yes
checking for readdir... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h
config.status: executing depfiles commands
make[1]: Entering directory 
`
Making all in compat
make[2]: Entering directory 
`
if /bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  -I../.. 
-I../../include -I../../src -I../include -I/usr/include/libxml2  
-I/usr/include/libxml2 -Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments 
 -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT assert.lo -MD -MP -MF 
".deps/assert.Tpo" -c -o assert.lo ../../compat/assert.cc; \
then mv -f ".deps/assert.Tpo" ".deps/assert.Plo"; else rm -f 
".deps/assert.Tpo"; exit 1; fi
if /bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  -I../.. 
-I../../include -I../../src -I../include -I/usr/include/libxml2  
-I/usr/include/libxml2 -Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments 
 -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT compat.lo -MD -MP -MF 
".deps/compat.Tpo" -c -o compat.lo ../../compat/compat.cc; \
then mv -f ".deps/compat.Tpo" ".deps/compat.Plo"; else rm -f 
".deps/compat.Tpo"; exit 1; fi
mkdir .libs
 g++ -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../include 
-I/usr/include/libxml2 -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
assert.lo -MD -MP -MF .deps/assert.Tpo -c ../../compat/assert.cc  -fPIC -DPIC 
-o .libs/assert.o
 g++ -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../include 
-I/usr/include/libxml2 -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
compat.lo -MD -MP -MF .deps/compat.Tpo -c ../../compat/compat.cc  -fPIC -DPIC 
-o .libs/compat.o
 g++ -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../include 
-I/usr/include/libxml2 -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
assert.lo -MD -MP -MF .deps/assert.Tpo -c ../../compat/assert.cc  -fPIC -DPIC 
-o assert.o >/dev/null 2>&1
 g++ -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../include 
-I/usr/include/libxml2 -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
compat.lo -MD -MP -MF .deps/compat.Tpo -c ../../compat/compat.cc  -fPIC -DPIC 
-o compat.o >/dev/null 2>&1
if /bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H  -I../.. 
-I../../include -I../../src -I../include -I/usr/include/libxml2  
-I/usr/include/libxml2 -Werror -Wall -Wpointer-arith -Wwrite-strings -Wcomments 
 -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT debug.lo -MD -MP -MF 
".deps/debug.Tpo" -c -o debug.lo ../../compat/debug.cc; \
then mv -f ".deps/debug.Tpo" ".deps/debug.Plo"; else rm -f 
".deps/debug.Tpo"; exit 1; fi
if /bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H  -I../.. 
-I../../include -I../../src -I../include -I/usr/include/libxml2  -Werror 
-Wall -Wpointer-arith -Wwrite-strings -Wmissing-prototypes 
-Wmissing-declarations -Wcomments -D_REENTRANT -D_FILE_OFFSET_BITS=64 -Wall -g 
-O2 -MT GnuRegex.lo -MD -MP -MF ".deps/GnuRegex.Tpo" -c -o GnuRegex.lo 
../../compat/GnuRegex.c; \
then mv -f ".deps/GnuRegex.Tpo" ".deps/GnuRegex.Plo"; else rm -f 
".deps/GnuRegex.Tpo"; exit 1; fi
 g++ -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../include 
-I/usr/include/libxml2 -I/usr/include/libxml2 -Werror -Wall -Wpointer-arith 
-Wwrite-strings -Wcomments -D_REENTRANT -D_FILE_OFFSET_BITS=64 -g -O2 -MT 
debug.lo -MD -MP -MF .deps/debug.Tpo -c ../../compat/debug.cc  -fPIC -DPIC -o 
.libs/debug.o
 gcc -DHAVE_CONFIG_H -I../.. -I../../include -I../../src -I../include 
-I/usr/include/libxml2 -Werror -Wall -Wpointer-arith -Wwrite-strings 
-Wmissing

Re: squid-smp

[Sachin Malave]

On Thu, Oct 15, 2009 at 5:56 AM, Adrian Chadd  wrote:
> Oh, I can absolutely give you guys food for thought. I was just hoping
> someone else would already try to do a bit of legwork.
>
> Things to think about:
>
> * Do you really, -really- want to reinvent the malloc wheel? This is
> separate from caching results and pre-made class instances. There's
> been a lot of work in well-performing, thread-aware malloc libraries
> * Do you want to run things in multiple processes or multiple threads?
> Or support both?
> * How much of the application do you want to push out into separate
> threads? run lots of "copies" of Squid concurrently, with some locking
> going on? Break up individual parts of the processing pipeline into
> threads? (eg, what I'm going to be experimenting with soon - handling
> ICP/HTCP in a separate thread for some basic testing)
> * Survey the current codebase and figure out what depends upon what -
> in a way that you can use for figuring out what needs to be made
> re-entrant and what may need locking. Think about how to achieve all
> of this. Best example of this - you're going to need to figure out how
> to do concurrent debug logging and memory allocation - so see what
> that code uses, what that codes' code uses, etc
> * 10GE cards are dumping individual PCIe channels to CPUs; which means
> that the "most efficient" way of pumping data around will be to
> somehow throw individual connections onto specific CPUs, and keep them
> there. There's no OS support for this yet, but OSes may be "magical"
> (ie, handing you sockets in specific threads via accept() and hoping
> that the NIC doesn't reorganise its connection->PCIe channel hash)
> * Do you think its worth being able to migrate specific connections
> between threads? Or once they're in a thread they're there for good
> * If you split up squid into "lots of threads running the whole app",
> what and where would you envisage locking and blocking? What about
> data sharing? How would that scale given a handful of example
> workloads? What about in abnormal situations? How well will things
> degrade?
> * What about using message passing and message queues? Where would it
> be appropriate? Where wouldn't it be appropriate? Why?
>
> Here's an example:
>
> * Imagine you're doing store lookups using message passing with your
> "store" being a separate thread with a message queue. Think about how
> you'd handle say, ICP peering between two caches doing > 10,000
> requests a second. What repercussions does that have for the locking
> of the message queues between other threads. What are the other
> threads doing?
>
> With that in mind, survey the kinds of ways that current network apps
> "do" threading:
>
> * look at the various ways apache does it - eg, the per-connection
> thread+process hybrid model, the event-worker thread model, etc

Yes, I think...  if we have loosely coupled
architecture(distributed or multiprocessor )  then it is better to use
processes otherwise on multi-core platform threading model can be
used ( I am targeting multi-core ).

> * look at memcached - one thread doing accept'ing, farming requests
> off to other threads that just run a squid-like event loop. Minimal
> inter-thread communication for the most part
> * investigate what the concurrency hooks for various frameworks do -
> eg, the boost asio library stuff has "colours" which you mark thread
> events with. These colours dictate which events need to be run
> sequentially and which can run in parallel
> * look at all of the random blogs written by windows networking coders
> - they're further ahead of the massively-concurrent network
> application stack because Windows has had it for a number of years.
>

Ok !!! Questions that you have raised would be considered while
creating threads or processes.

> Now. You've mentioned you've looked at the others and you think major
> replumbing is going to be needed. Here's a hint - its going to be
> needed. Thinking you can avoid it is silly. Figuring out what you can
> do right now that doesn't lock you into a specific trajectory is -not-
> silly. For example, figuring out what APIs need to be changed to make
> them re-enterant is not silly. Most of the stuff in lib/ with static
> char buffers that they return need to be changed. That can be done
> -now- without having to lock yourself into a particular concurrency
> model.
>
> 2c,


thank you :)

>
>
>
> Adrian
>
> 2009/10/15 Amos Jeffries :
>> Adrian Chadd wrote:
>>>
>>> 2009/10/15 Sachin Malave :
>>>
 Its not like we want to make project bad. Squid was not deployed on
 smp before because we did not have shared memory architectures
 (multi-cores), also the library support for multi-threading was like
 nightmare for people. Now things are changed, it is very easy to
 manage threads, people have multi-core machines at their desktops, and
 as hardware is available now or later somebody has to try and build
 

Re: squid-smp

[Adrian Chadd]

Oh, I can absolutely give you guys food for thought. I was just hoping
someone else would already try to do a bit of legwork.

Things to think about:

* Do you really, -really- want to reinvent the malloc wheel? This is
separate from caching results and pre-made class instances. There's
been a lot of work in well-performing, thread-aware malloc libraries
* Do you want to run things in multiple processes or multiple threads?
Or support both?
* How much of the application do you want to push out into separate
threads? run lots of "copies" of Squid concurrently, with some locking
going on? Break up individual parts of the processing pipeline into
threads? (eg, what I'm going to be experimenting with soon - handling
ICP/HTCP in a separate thread for some basic testing)
* Survey the current codebase and figure out what depends upon what -
in a way that you can use for figuring out what needs to be made
re-entrant and what may need locking. Think about how to achieve all
of this. Best example of this - you're going to need to figure out how
to do concurrent debug logging and memory allocation - so see what
that code uses, what that codes' code uses, etc
* 10GE cards are dumping individual PCIe channels to CPUs; which means
that the "most efficient" way of pumping data around will be to
somehow throw individual connections onto specific CPUs, and keep them
there. There's no OS support for this yet, but OSes may be "magical"
(ie, handing you sockets in specific threads via accept() and hoping
that the NIC doesn't reorganise its connection->PCIe channel hash)
* Do you think its worth being able to migrate specific connections
between threads? Or once they're in a thread they're there for good
* If you split up squid into "lots of threads running the whole app",
what and where would you envisage locking and blocking? What about
data sharing? How would that scale given a handful of example
workloads? What about in abnormal situations? How well will things
degrade?
* What about using message passing and message queues? Where would it
be appropriate? Where wouldn't it be appropriate? Why?

Here's an example:

* Imagine you're doing store lookups using message passing with your
"store" being a separate thread with a message queue. Think about how
you'd handle say, ICP peering between two caches doing > 10,000
requests a second. What repercussions does that have for the locking
of the message queues between other threads. What are the other
threads doing?

With that in mind, survey the kinds of ways that current network apps
"do" threading:

* look at the various ways apache does it - eg, the per-connection
thread+process hybrid model, the event-worker thread model, etc
* look at memcached - one thread doing accept'ing, farming requests
off to other threads that just run a squid-like event loop. Minimal
inter-thread communication for the most part
* investigate what the concurrency hooks for various frameworks do -
eg, the boost asio library stuff has "colours" which you mark thread
events with. These colours dictate which events need to be run
sequentially and which can run in parallel
* look at all of the random blogs written by windows networking coders
- they're further ahead of the massively-concurrent network
application stack because Windows has had it for a number of years.

Now. You've mentioned you've looked at the others and you think major
replumbing is going to be needed. Here's a hint - its going to be
needed. Thinking you can avoid it is silly. Figuring out what you can
do right now that doesn't lock you into a specific trajectory is -not-
silly. For example, figuring out what APIs need to be changed to make
them re-enterant is not silly. Most of the stuff in lib/ with static
char buffers that they return need to be changed. That can be done
-now- without having to lock yourself into a particular concurrency
model.

2c,



Adrian

2009/10/15 Amos Jeffries :
> Adrian Chadd wrote:
>>
>> 2009/10/15 Sachin Malave :
>>
>>> Its not like we want to make project bad. Squid was not deployed on
>>> smp before because we did not have shared memory architectures
>>> (multi-cores), also the library support for multi-threading was like
>>> nightmare for people. Now things are changed, it is very easy to
>>> manage threads, people have multi-core machines at their desktops, and
>>> as hardware is available now or later somebody has to try and build
>>> SMP support. think about future...
>>>
>>> To cop with internet speed & increase in number of users, Squid must
>>> use multi-core architecture and distribute its work
>>
>> I 100% agree with your comments. I agree 100% that Squid needs to be
>> made scalable on multi-core boxes.
>>
>> Writing threaded code may be easier now than in the past, but the ways
>> of screwing stability, debuggability, performance and such -haven't-
>> changed.. This is what I'm trying to get across. :)
>
> Aye, understood. Which is why I've made sure all this discussion is done i

Re: squid-smp

[Amos Jeffries]

Adrian Chadd wrote:

2009/10/15 Sachin Malave :


Its not like we want to make project bad. Squid was not deployed on
smp before because we did not have shared memory architectures
(multi-cores), also the library support for multi-threading was like
nightmare for people. Now things are changed, it is very easy to
manage threads, people have multi-core machines at their desktops, and
as hardware is available now or later somebody has to try and build
SMP support. think about future...

To cop with internet speed & increase in number of users, Squid must
use multi-core architecture and distribute its work


I 100% agree with your comments. I agree 100% that Squid needs to be
made scalable on multi-core boxes.

Writing threaded code may be easier now than in the past, but the ways
of screwing stability, debuggability, performance and such -haven't-
changed.. This is what I'm trying to get across. :)


Aye, understood. Which is why I've made sure all this discussion is done 
in squid-dev. So those like yourself who might have anything to point at 
as good/bad examples can do so.


Sure, Squid can be re-written from the group up yet again. But none of 
us want the ten year delay that will cause. The answer is to drop eight 
years of improvements and use the Squid-2 code, or go ahead with a 
somewhat incompletely upgraded Squid-3 code. Leveraging some of the SMP 
work to further upgrade the remaining sections, while just slipping SMP 
into the currently upgraded components.


Do you actually have any relevant implementations you in your infinite 
wisdom and foresight want to point us at? Or just diss us for not 
knowing enough?


I'm already aware of the overall models Varnish, Oops, Apache, and 
Polipo, and Nginx are documented as using. Without looking at the code 
it's clear that their approaches are not beneficial to Squid without 
major re-plumbing.


The solution we have to use is a mix, possibly unique to Squid, which 
retains Squids features and niche coverage. The right mix of tools for 
each task to be performed: child processes, IPC, and events. Now adding 
threads for the pieces that are applicable. There is order in the chaos.


Amos
--
Please be using
  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19
  Current Beta Squid 3.1.0.14

Re: squid-smp

[Adrian Chadd]

2009/10/15 Sachin Malave :

> Its not like we want to make project bad. Squid was not deployed on
> smp before because we did not have shared memory architectures
> (multi-cores), also the library support for multi-threading was like
> nightmare for people. Now things are changed, it is very easy to
> manage threads, people have multi-core machines at their desktops, and
> as hardware is available now or later somebody has to try and build
> SMP support. think about future...
>
> To cop with internet speed & increase in number of users, Squid must
> use multi-core architecture and distribute its work

I 100% agree with your comments. I agree 100% that Squid needs to be
made scalable on multi-core boxes.

Writing threaded code may be easier now than in the past, but the ways
of screwing stability, debuggability, performance and such -haven't-
changed.. This is what I'm trying to get across. :)



Adrian

Re: squid-smp

[Sachin Malave]

On Wed, Oct 14, 2009 at 11:17 PM, Adrian Chadd  wrote:
> 2009/10/14 Amos Jeffries :
>
> [snip]
>
> I still find it very amusing that noone else has sat down and talked
> about the last 20 + years of writing threaded, concurrent code and
> what the pro/cons of them would be here; nor what other projects are
> doing.
>
> Please don't sit down and talk about how to shoehorn SMP into some
> existing Squid-3 "thing" (be it AsyncCalls, or anything really) before
> doing this. You'll just be re-inventing the same mistakes made in the
> past and it will make the project look bad.
>
>
>
> Adrian
>

Its not like we want to make project bad. Squid was not deployed on
smp before because we did not have shared memory architectures
(multi-cores), also the library support for multi-threading was like
nightmare for people. Now things are changed, it is very easy to
manage threads, people have multi-core machines at their desktops, and
as hardware is available now or later somebody has to try and build
SMP support. think about future...

To cop with internet speed & increase in number of users, Squid must
use multi-core architecture and distribute its work



-- 
Mr. S. H. Malave
Computer Science & Engineering Department,
Walchand College of Engineering,Sangli.
sachinmal...@wce.org.in