best practices for log rotation

[Charles Allen]

Anyone have good hints for best practices for log rotation?

Our mesos master ended up with many gigabytes of logs once we started
running SPARK on it (approx 2GB of master INFO logs per day).

Re: mesos cpuset isolator module available

[Charles Allen]

Super cool! does this also support limiting which memory controller the
tasks can use?

On Thu, Dec 15, 2016 at 10:00 AM Zhitao Li  wrote:

> Thanks for sharing. This is very interesting to us because we are also
> looking for solution for latency sensitive CPU isolation.
>
> On Thu, Dec 15, 2016 at 9:56 AM, ct clmsn  wrote:
>
> I'll add in BUILD instructions tonight/this weekend. I'll be releasing
> some performance counter tools to use in a mesos system (for container
> applications) very soon.
>
> On Thu, Dec 15, 2016 at 12:13 PM, tommy xiao  wrote:
>
> thanks for your sharing. v5
>
> 2016-12-15 23:40 GMT+08:00 ct clmsn :
>
> I've completed a mesos module to support cgroups cpusets. This work is
> related to a JIRA ticket that I posted last spring (MESOS-5342). Apologies
> for the long delay wrapping up the implementation.
>
> https://github.com/ct-clmsn/mesos-cpusets
>
> If you test it out, have issues, or want to make improvements, please post
> to github - I've done some very simple/trivial testing.
>
> Chris
>
>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
>
>
>
>
>
> --
> Cheers,
>
> Zhitao Li
>

Re: Multi-agent machine

[Charles Allen]

Ok, thanks!

On Fri, Dec 9, 2016 at 2:32 PM Benjamin Mahler <bmah...@apache.org> wrote:

> Maintenance should work in this case, it will just be applied to all
> agents on the machine.
>
> On Fri, Dec 9, 2016 at 1:20 PM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
> Thanks for the insight.
>
> I take that to mean the maintenance primitives might not work right for
> multi-agent machines? aka, I can't do maintenance on one agent but not the
> others?
>
>
> On Fri, Dec 9, 2016 at 12:16 PM Jie Yu <yujie@gmail.com> wrote:
>
> Charles,
>
> It should be possible. Here are the global 'object' that might conflict:
> 1) cgroup (you can use different cgroup root)
> 2) work_dir and runime_dir (you can set them to be different between
> agents)
> 3) network (e.g., iptables, if you use host network, should not be a
> problem. Otherwise, you might need to configure your network isolator
> properly)
>
> But we haven't tested. Another potential thing that might come up is the
> code that rely on hostname of the agent (MachineID in maintenance
> primitive?)
>
> - Jie
>
> On Fri, Dec 9, 2016 at 12:11 PM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
> Is it possible to setup a machine such that multiple mesos agents are
> running on the same machine and registering with the same master?
>
> For example, with different cgroup roots or different default working
> directory.
>
>
>
>

Re: Multi-agent machine

[Charles Allen]

Thanks for the insight.

I take that to mean the maintenance primitives might not work right for
multi-agent machines? aka, I can't do maintenance on one agent but not the
others?


On Fri, Dec 9, 2016 at 12:16 PM Jie Yu <yujie@gmail.com> wrote:

> Charles,
>
> It should be possible. Here are the global 'object' that might conflict:
> 1) cgroup (you can use different cgroup root)
> 2) work_dir and runime_dir (you can set them to be different between
> agents)
> 3) network (e.g., iptables, if you use host network, should not be a
> problem. Otherwise, you might need to configure your network isolator
> properly)
>
> But we haven't tested. Another potential thing that might come up is the
> code that rely on hostname of the agent (MachineID in maintenance
> primitive?)
>
> - Jie
>
> On Fri, Dec 9, 2016 at 12:11 PM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
> Is it possible to setup a machine such that multiple mesos agents are
> running on the same machine and registering with the same master?
>
> For example, with different cgroup roots or different default working
> directory.
>
>
>

Multi-agent machine

[Charles Allen]

Is it possible to setup a machine such that multiple mesos agents are
running on the same machine and registering with the same master?

For example, with different cgroup roots or different default working
directory.

Re: Unified cgroups isolator

[Charles Allen]

That's awesome, great work!


On Tue, Sep 13, 2016 at 5:10 PM Qian Zhang  wrote:

> Thanks @haosdent's awesome work and @Jie's great shepherding and guidance
> on this project!
>
>
> Thanks,
> Qian Zhang
>
> On Wed, Sep 14, 2016 at 7:56 AM, Gilbert Song 
> wrote:
>
>> Awesome!
>>
>> Kudos to @haosdent and @qianzhang!
>>
>> On Tue, Sep 13, 2016 at 11:22 AM, haosdent  wrote:
>>
>>> Really appreciate @qian and @jie's great helps on this! It makes us
>>> easier to add cgroups isolation for rest subsystem.
>>>
>>> Additionally, if you find any changes about unified cgroups isolator
>>> break your environment, please let us know. I would
>>> try to fix asap.
>>>
>>> On Wed, Sep 14, 2016 at 1:59 AM, Jie Yu  wrote:
>>>
 Hi,

 We just merged the unified cgroups isolator. Huge shout out to
 @haosdent and @qianzhang to make this happen!
 https://issues.apache.org/jira/browse/MESOS-4697

 Just to give you some context. Previously, it's a huge pain to add a
 new cgroups subsystem to Mesos because it requires creating a new isolator
 (a lot of code duplication). Now, we merge all the subsystems into one
 single isolator, that makes adding a new subsystem very easy.

 More importantly, the new cgroups isolator supports cgroups v2!

 - Jie

>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Haosdent Huang
>>>
>>
>>
>

Re: cgroup limits on memory page cache

[Charles Allen]

Of course, after sending out the email I thought of a potential solution.
It should be possible to specify memory available per role that is more
than the total system memory. So as long as the role understands that a
large swath of the available memory is intended to be page cache, all
should be good... I think

On Tue, Aug 9, 2016 at 8:16 PM Charles Allen <charles.al...@metamarkets.com>
wrote:

> I have two services I want to make co-tenant on the same machines, where
> one has much more relaxed SLAs than the other.
>
> I can easily make one pre-empt the other at a cgroup level by controlling
> their cpu shares.
>
> And I can limit one by having memory limits assignable through its role
> resources.
>
> The other higher priority service makes heavy use of disk page cache, and
> therefore would like to use as much memory for disk page cache as possible,
> but allow the lower-priority service to gobble up SOME of that page cache
> (to a limit)
>
> This is achievable with cgroups alone by setting appropriate cpu shares
> between the cgroups, and setting the memory limit of one cgroup to
> unlimited, while bounding the memory limit of the lesser priority service.
>
> Can I achieve such a thing in Mesos role resource definitions?
>

cgroup limits on memory page cache

[Charles Allen]

I have two services I want to make co-tenant on the same machines, where
one has much more relaxed SLAs than the other.

I can easily make one pre-empt the other at a cgroup level by controlling
their cpu shares.

And I can limit one by having memory limits assignable through its role
resources.

The other higher priority service makes heavy use of disk page cache, and
therefore would like to use as much memory for disk page cache as possible,
but allow the lower-priority service to gobble up SOME of that page cache
(to a limit)

This is achievable with cgroups alone by setting appropriate cpu shares
between the cgroups, and setting the memory limit of one cgroup to
unlimited, while bounding the memory limit of the lesser priority service.

Can I achieve such a thing in Mesos role resource definitions?

Re: Master pailer failure 0.28.2

[Charles Allen]

Slave pailer is working fine

On Tue, Aug 9, 2016 at 10:29 AM Charles Allen <charles.al...@metamarkets.com>
wrote:

> For some reason I started getting the following failure on 0.28.2 with the
> pailer when trying to view master logs from the master console (REDACTED is
> the ip address):
>
>
> angular-1.2.3.min.js:84 Error: [$interpolate:interr]
> http://errors.angularjs.org/1.2.3/$interpolate/interr?p0=%7B%7Boffered_cpus
> …7D=TypeError%3A%20Cannot%20read%20property%20'toFixed'%20of%20undefined
> at Error (native)
> at http://REDACTED:5050/static/js/angular-1.2.3.min.js:6:449
> at Object.s (
> http://REDACTED:5050/static/js/angular-1.2.3.min.js:73:495)
> at f.$digest (
> http://REDACTED:5050/static/js/angular-1.2.3.min.js:99:14)
> at f.$apply (
> http://REDACTED:5050/static/js/angular-1.2.3.min.js:101:369)
> at f (http://REDACTED:5050/static/js/angular-1.2.3.min.js:67:175)
> at Q (http://REDACTED:5050/static/js/angular-1.2.3.min.js:71:99)
> at XMLHttpRequest.y.onreadystatechange (
> http://REDACTED:5050/static/js/angular-1.2.3.min.js:72:130)(anonymous
> function) @ angular-1.2.3.min.js:84(anonymous function) @
> angular-1.2.3.min.js:62s @ angular-1.2.3.min.js:74$digest @
> angular-1.2.3.min.js:99$apply @ angular-1.2.3.min.js:101f @
> angular-1.2.3.min.js:67Q @ angular-1.2.3.min.js:71y.onreadystatechange @
> angular-1.2.3.min.js:72
> angular-1.2.3.min.js:84 Error: [$interpolate:interr]
> http://errors.angularjs.org/1.2.3/$interpolate/interr?p0=%7B%7Bidle_cpus%20
> …7D=TypeError%3A%20Cannot%20read%20property%20'toFixed'%20of%20undefined
> at Error (native)
> at http://REDACTED:5050/static/js/angular-1.2.3.min.js:6:449
> at Object.s (
> http://REDACTED:5050/static/js/angular-1.2.3.min.js:73:495)
> at f.$digest (
> http://REDACTED:5050/static/js/angular-1.2.3.min.js:99:14)
> at f.$apply (
> http://REDACTED:5050/static/js/angular-1.2.3.min.js:101:369)
> at f (http://REDACTED:5050/static/js/angular-1.2.3.min.js:67:175)
> at Q (http://REDACTED:5050/static/js/angular-1.2.3.min.js:71:99)
> at XMLHttpRequest.y.onreadystatechange (
> http://REDACTED:5050/static/js/angular-1.2.3.min.js:72:130)(anonymous
> function) @ angular-1.2.3.min.js:84(anonymous function) @
> angular-1.2.3.min.js:62s @ angular-1.2.3.min.js:74$digest @
> angular-1.2.3.min.js:99$apply @ angular-1.2.3.min.js:101f @
> angular-1.2.3.min.js:67Q @ angular-1.2.3.min.js:71y.onreadystatechange @
> angular-1.2.3.min.js:72
>
>
> Has anyone seen this before?
>

Re: what's the pronunciation of "MESOS"?

[Charles Allen]

My wife thought I was crazy sitting here mumbling "mAY-sohs" "MEH-sohs"
"Mee-sohs"

On Mon, Aug 8, 2016 at 9:22 PM Yu Wei  wrote:

> Thanks Joe.
>
> It's really interesting.
>
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>
>
> --
> *From:* Joseph Jacks 
> *Sent:* Tuesday, August 9, 2016 10:53 AM
> *To:* user@mesos.apache.org
> *Subject:* Re: what's the pronunciation of "MESOS"?
>
> "MAY-zoss" is most common and correct.
>
> "MEH-zoss" is second most common and also correct, I think.
>
> "MEE-zoss" is third most common, but incorrect.
>
> JJ.
>
> On Aug 8, 2016, at 10:48 PM, Yu Wei  wrote:
>
>
> Thx,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>
>

Master pailer failure 0.28.2

[Charles Allen]

For some reason I started getting the following failure on 0.28.2 with the
pailer when trying to view master logs from the master console (REDACTED is
the ip address):


angular-1.2.3.min.js:84 Error: [$interpolate:interr]
http://errors.angularjs.org/1.2.3/$interpolate/interr?p0=%7B%7Boffered_cpus
…7D=TypeError%3A%20Cannot%20read%20property%20'toFixed'%20of%20undefined
at Error (native)
at http://REDACTED:5050/static/js/angular-1.2.3.min.js:6:449
at Object.s (http://REDACTED:5050/static/js/angular-1.2.3.min.js:73:495)
at f.$digest (http://REDACTED:5050/static/js/angular-1.2.3.min.js:99:14)
at f.$apply (http://REDACTED:5050/static/js/angular-1.2.3.min.js:101:369
)
at f (http://REDACTED:5050/static/js/angular-1.2.3.min.js:67:175)
at Q (http://REDACTED:5050/static/js/angular-1.2.3.min.js:71:99)
at XMLHttpRequest.y.onreadystatechange (
http://REDACTED:5050/static/js/angular-1.2.3.min.js:72:130)(anonymous
function) @ angular-1.2.3.min.js:84(anonymous function) @
angular-1.2.3.min.js:62s @ angular-1.2.3.min.js:74$digest @
angular-1.2.3.min.js:99$apply @ angular-1.2.3.min.js:101f @
angular-1.2.3.min.js:67Q @ angular-1.2.3.min.js:71y.onreadystatechange @
angular-1.2.3.min.js:72
angular-1.2.3.min.js:84 Error: [$interpolate:interr]
http://errors.angularjs.org/1.2.3/$interpolate/interr?p0=%7B%7Bidle_cpus%20
…7D=TypeError%3A%20Cannot%20read%20property%20'toFixed'%20of%20undefined
at Error (native)
at http://REDACTED:5050/static/js/angular-1.2.3.min.js:6:449
at Object.s (http://REDACTED:5050/static/js/angular-1.2.3.min.js:73:495)
at f.$digest (http://REDACTED:5050/static/js/angular-1.2.3.min.js:99:14)
at f.$apply (http://REDACTED:5050/static/js/angular-1.2.3.min.js:101:369
)
at f (http://REDACTED:5050/static/js/angular-1.2.3.min.js:67:175)
at Q (http://REDACTED:5050/static/js/angular-1.2.3.min.js:71:99)
at XMLHttpRequest.y.onreadystatechange (
http://REDACTED:5050/static/js/angular-1.2.3.min.js:72:130)(anonymous
function) @ angular-1.2.3.min.js:84(anonymous function) @
angular-1.2.3.min.js:62s @ angular-1.2.3.min.js:74$digest @
angular-1.2.3.min.js:99$apply @ angular-1.2.3.min.js:101f @
angular-1.2.3.min.js:67Q @ angular-1.2.3.min.js:71y.onreadystatechange @
angular-1.2.3.min.js:72


Has anyone seen this before?

Re: Mesos logging

[Charles Allen]

Which glog are you trying to capture? You can set the verbosity level with
the environment variable GLOG_v

And you can also set it through things like Spark. So if you want a lot of
ZK chatter at the mesos level in your spark logs, add

spark.executorEnv.GLOG_v=9

to your spark context

On Mon, Aug 8, 2016 at 2:53 PM Hendrik Haddorp 
wrote:

> Hi,
>
> the Mesos code contains log statements using LOG(INFO) and VLOG(1), for
> example. So far I found that Mesos is using the Google Logging Library.
> Looking in the logs I only seem to be able to find output from VLOG
> statements. What do I need to do to get the output from the LOG
> statements? Where would I typically find the output? I'm using CentOS
> 7.2 and found the output so far in the files below /var/log/mesos.
>
> thanks,
> Hendrik
>

On a whim I sync'd my local master branch

[Charles Allen]

And ran the unit tests (through jenkins)

[--] Global test environment tear-down
[==] 1209 tests from 140 test cases ran. (360352 ms total)
[  PASSED  ] 1209 tests.

  YOU HAVE 6 DISABLED TESTS

Finished: *SUCCESS*


That's a result I hadn't seen in a few days. Good to see the progress,
thanks mesos team

Re: Completely rude alternative to HierarchicalDRF?

[Charles Allen]

Thanks Benjamin, that's quite insightful and also a bummer.

On Fri, Jan 29, 2016 at 11:50 AM Benjamin Mahler <bmah...@apache.org> wrote:

> Hey Charles,
>
> From this question it seems you're aware that the default allocator can be
> swapped out for alternative implementations. However, the allocator module
> interface is wide currently and so you'll be hard-pressed to find
> implementations. (No refactoring took place during modularization).
> Implementations will have had to keep up with a number of additional
> responsibilities over time (generating inverse offers, handling quota,
> etc). I'm thinking we'll need to figure out which pieces of functionality
> *within* the allocator should be made modularizable when we do some of the
> upcoming allocation work. For now, the expectation is that allocator module
> implementations are non-trivial to maintain.
>
> Hope that helps!
>
> On Mon, Jan 25, 2016 at 3:18 PM, Charles Allen <
> charles.al...@metamarkets.com> wrote:
>
>> Is there an allocator out there that will prioritize frameworks to favor
>> the oldest first?
>>
>> I'm using spark in coarse mode, and I actually want the earliest
>> registered course mode drivers to get as many resources as they want.
>>
>
>

GLOG settings

[Charles Allen]

Just a general FYI, you can tune logging settings by using the GLOG_ prefix
to environment variables. For example GLOG_v=9 turns up the verbosity
level. This works if you're running a java framework with the native java
library. Unfortunately, I haven't found a way to plug GLOG --> SLF4J

Re: Native Lib vs. Rest API

[Charles Allen]

@Ben : Is the status of the mesos-rxjava MORE or LESS functional than the
legacy mesos .jar with the native library?

On Thu, Dec 3, 2015 at 12:05 PM Ben Whitehead 
wrote:

> Hi John,
>
> If you're using Java there is already a prototype client atop the new
> Scheduler HTTP API using RxJava:
> https://github.com/mesosphere/mesos-rxjava Happy to provide more info if
> interested.
>
> --Ben Whitehead
>
> On Thu, Dec 3, 2015 at 11:52 AM, John Omernik  wrote:
>
>> Thank you!
>>
>> On Thu, Dec 3, 2015 at 1:40 PM, Vinod Kone  wrote:
>>
>>> Yes, that's the plan.
>>>
>>> Here are the related epics tracking the work: MESOS-2288
>>>  and MESOS-3302
>>> 
>>>
>>> The user doc for the scheduler API is
>>> https://github.com/apache/mesos/blob/master/docs/scheduler-http-api.md
>>>
>>> On Thu, Dec 3, 2015 at 11:34 AM, John Omernik  wrote:
>>>
 Somewhere in the back of my brain I thought I read something about a
 migration away from using the mesos native lib and going to a more generic
 API approach to support better portability and less reliance on the lib.

 I read about this before I understood things well (or as well I do now
 I should say). Am I misremembering reading about this? I can't find any
 stories/documentation on this. If I am correct on this, can someone point
 me to a JIRA or a discussion on how this is supposed to work? I.e. is the
 goal to migrate all frameworks off the native library to deprecate it etc?

 Thanks, sorry for the weird questions.

 John

>>>
>>>
>>
>

ipv6 resource or attributes

[Charles Allen]

>From just a cursory glance at
http://mesos.apache.org/documentation/attributes-resources/ I'm curious if
ipv6 addresses can be used as a text resource or attribute.

I have not experimented with this at all yet, and was wondering if such use
cases are intended to be supported.

It seems that it might be possible to use an ipv4 address as the text of a
set attribute

some_key:{127.0.0.1}

but not as a distinct text resource or attribute

some_key:127.0.0.1

as that would collide with the float specification in a strict
interpretation

But it seems that an ipv6 address like 2001:db8::ff00:42:8329  wold cause
problems both as a distinct text attirbute/resource

some_key:2001:db8::ff00:42:8329

or as part of a set

some_key:{2001:db8::ff00:42:8329}

because `:' is not listed among the allowed characters for text.

Is this correct reading of the attributes and resources specification?

Thanks,
Charles Allen

Re: Resource overcommittal

[Charles Allen]

I did experiments and a minimal amount of surface research on this and
found out the following items at least tangentially related to OPs question:

   1. CGroups treats paged-in memory mapped files TOWARDS your memory
   limit, even if they are mapped shared. I have not yet tested what happens
   if tasks from different cgroups map the same file, but it *might* over
   account memory usage in such a case.
   2. CPUs really stands for CPUshares. (assuming you don't enable CFS
   limiting) If you give a task 0.1 or 1M CPU shares it won't make a
   difference until the resource is under contention, only then will the CFS
   enforce the share ratio.
   3. Linux kernel has overcommitting of memory
   https://www.kernel.org/doc/Documentation/vm/overcommit-accounting  but I
   have not experimented with tweaking this setting and adjusting mesos
   resources.

Cheers,
Charles Allen

On Sat, Apr 11, 2015 at 1:41 AM, Adam Bordelon a...@mesosphere.io wrote:

 See also the Optimistic Offers JIRA:
 https://issues.apache.org/jira/browse/MESOS-1607

 On Fri, Apr 10, 2015 at 12:21 PM, Benjamin Mahler 
 benjamin.mah...@gmail.com wrote:

 There is a really stale one here:
 https://issues.apache.org/jira/browse/MESOS-354

 It's either going to get converted to an epic, or we can replace it with
 a fresh epic with a design and eventually a work breakdown. Add yourself to
 the watchers for updates.

 On Fri, Apr 10, 2015 at 12:03 PM, Tom Arnfeld t...@duedil.com wrote:

 Is there a ticket for this specific bit of work, Ben?

 --

 Tom Arnfeld
 Developer // DueDil

 (+44) 7525940046
 25 Christopher Street, London, EC2A 2BS


 On Fri, Apr 10, 2015 at 6:28 PM, Eduardo Jiménez yoeduar...@gmail.com
 wrote:

 Hi,

 Does mesos today overcommit server resources if the current offers that
 have been accepted are not being used?

 I have cpu/mem cgroup isolators turned on, and I see the cluster
 running out of resources to offer, even if the applications accepting the
 current offers aren't fully using them. I'm wondering if choosing to enable
 the isolators causes this behavior.

 Thanks,

 Eduardo

Re: SharedFilesystem and rolling restarts

[Charles Allen]

That looks like it would solve it, thanks!

On Tue, Mar 31, 2015 at 4:24 PM, Jie Yu yujie@gmail.com wrote:

 Hi, in Mesos 0.23.0 (the next release), you'll be able to use the
 persistence primitives provided by Mesos.
 https://issues.apache.org/jira/browse/MESOS-1554

 Basically, your framework can create a persistent volume (has a unique
 handle) while launching a task and re-use that handle when re-launching the
 task. The same persistent volume will be mounted into the sandbox. Any data
 stored in the volume will be persisted.

 - Jie

 On Tue, Mar 31, 2015 at 4:00 PM, Charles Allen 
 charles.al...@metamarkets.com wrote:

 I am working on potentially porting druid.io to a mesos framework. One
 limitation for production use is that there is a lot of data cached locally
 on disk that does not need to be re-fetched during a rolling restart.

 If I were to take the simplest mesos route, each instance of the
 disk-cache-heavy task would have its own executor and would have to refresh
 the disk cache from deep storage each time it starts.

 A more complex route would be to have a standalone executor which handles
 the forking and restarts of tasks in order to maintain the working
 directory of the task.

 A slightly more hacky way of doing it would be to allow the
 disk-cache-heavy task to each have their own executor but use a common
 SharedFilesystem. But I'm not clear if SharedFilesystem would persist
 beyond an executor's lifespan.

 In such a case (where on-disk data would need to be immediately
 available after a rolling restart) is there a recommended approach to
 making sure the data persists properly?

 Thanks,
 Charles Allen






-- 

http://www.metamarkets.com/ Charles Allen PhDSr. Software Engineer|
METAMARKETS http://www.metamarkets.com/m 765.490.0454|t @drcrallen
https://twitter.com/drcrallencharles.al...@metamarkets.com

SharedFilesystem and rolling restarts

[Charles Allen]

I am working on potentially porting druid.io to a mesos framework. One
limitation for production use is that there is a lot of data cached locally
on disk that does not need to be re-fetched during a rolling restart.

If I were to take the simplest mesos route, each instance of the
disk-cache-heavy task would have its own executor and would have to refresh
the disk cache from deep storage each time it starts.

A more complex route would be to have a standalone executor which handles
the forking and restarts of tasks in order to maintain the working
directory of the task.

A slightly more hacky way of doing it would be to allow the
disk-cache-heavy task to each have their own executor but use a common
SharedFilesystem. But I'm not clear if SharedFilesystem would persist
beyond an executor's lifespan.

In such a case (where on-disk data would need to be immediately available
after a rolling restart) is there a recommended approach to making sure the
data persists properly?

Thanks,
Charles Allen