I consider that the most strenuous work is mainly focused on staying as
informed as possible about vulnerabilities in critical applications (kernel,
web browser, etc.), although it is not too difficult to compile a new kernel
or a new version of the web browser over time becomes very tedious or
demanding to follow the do it your self philosophy because you would spend
more time worrying about security breaches or in finding a hotfix for an old
version of a dependency than having fun with your device. Remember that you
will be on your own and the time it usually takes to compile a new version of
the kernel or web browser is quite considerable (one to two hours at best and
with high-end equipment) task that as a minimum should be performed once a
month in the best of cases and in the worst could be 2 to 3 times in a month
per application.
My intention is not to discourage you, but it's a little more acceptable when
you have some packers or developers covering your back and take from them the
most time-consuming applications to compile.
Best regards.