Re: Can anyone make suggestions for training / learning resources wrt databases.

[Russell Coker via luv-main]

On Sunday, 18 September 2016 10:54:06 AM AEST h via luv-main wrote:
> *  Writing sql / script files
> *  Structuring tables either within a database, or on a server

These seem like good topics for hands-on training at the Beginners' SIG 
meetings.  H can you make it to those meetings?  Would anyone like to run a 
training session on one of those topics?

-- 
My Main Blog http://etbe.coker.com.au/
My Documents Bloghttp://doc.coker.com.au/

___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

Re: Can anyone make suggestions for training / learning resources wrt databases.

[Trish Fraser via luv-main]

Hi Hugh,

> I am a linux hobbyist.  
> I have been using postgis for a while, at a hobby level,
> to provide data for web pages and for QGIS.

Are you using psql, or some other means for scripting?

Postgis appears to be an extender for Postgres, so you could try adding
Postgres to your search queries.

Cheers!

-- 
Trish Fraser, VVMZ4 91L2V -35.67910, 142.66607
Sun Sep 18 13:30:02 AEST 2016
GNU/Linux 1997-2013 #283226 counter.li.org
cassiopeia up up 5 days, 20 hours, 3 minutes
Mageia release 5 (Official) for x86_64
kernel 4.4.16-desktop-1.mga5


pgpfjxBILot60.pgp
Description: OpenPGP digital signature
___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

Re: Can anyone make suggestions for training / learning resources wrt databases.

[Craig Sanders via luv-main]

On Sun, Sep 18, 2016 at 10:54:06AM +1000, h wrote:
> There are two areas I have been unable to find information on:
> 
> *  Writing sql / script files

An sql script file is just a bunch of sql commands in a sequence.

There are numerous ways to run such a script, including piping or
redirecting it into /usr/bin/psql, using `\i scriptname.sql` from within
an existing psql session, writing a simple perl DBI wrapper (or similar
in python or whatever) to connect to the DB and start issuing SQL
commands, etc.

The latter is most useful if you want conditional execution of commands
depending on the results of previous commands, but don't want to write
in a db-specific language like pl/sql (note: postgresql, at least, has
options to embed the perl, python, lua, sh, and/or tcl languages into
the postgres server itself so you can write stored procedures in those
languages too).

It's usually best to run sql scripts wrapped in a transaction, so that
either all of the commands in the script succeed or they all get rolled
back as if they'd never happened.

What else do you need to know?

> *  Structuring tables either within a database, or on a server

That's a much more complex topic.  There's no set answer, it depends
entirely on your data and how you intend to use it.

Wikipedia has a set of articles on this topic, which serve as a great
introduction to the concepts.  I'd start with:

https://en.wikipedia.org/wiki/Database_normalization

Read that, and then follow your interests with the links at the bottom.
It won't tell you everything you need to know, but a few hours reading
will at least teach you enough to know what to search for.

Stack Exchange (SE) also has a site dedicated to db-related questions
and answers, at http://dba.stackexchange.com/

http://dba.stackexchange.com/help/on-topic says:

dba.se is for those needing expert answers to advanced database-related
questions concerning traditional SQL RDBMS and NoSQL alternatives.

If you have a question about...

* Database Administration including configuration and backup / restore
* Advanced Querying including window-functions, dynamic-sql, and 
  query-performance
* Data Modelling and database-design, including referential-integrity
* Advanced Programming in built-in server-side languages including
  stored-procedures and triggers.
* Data Warehousing and Business Intelligence including etl, reporting,
  and olap

...then you're in the right place to ask your question!


Even if you don't post a question yourself, there are lots of good
questions and answers in an easy to find format (unlike a forum site,
you won't have to wade through page after page of inconsequential chat,
bickering, ill-informed nonsense, obsolete information etc just to find
the few hidden gems of useful information)



> The particular problems I have are:
> 
> * I regularly update my tables from multiple csv files, all residing
>   in the same folder. Currently I have a script with a hardwired path
>   for each csv file. I would like to have a single 'variable' I could
>   change to define the path to all the csvs.

No matter which language the script is written in, the best solution
for this is to use getopts to process command line option.  Even in sh
or bash, it's a lot easier than you might think to get good option and
argument processing, just like "real programs" :)

e.g. your script could have a '-p pathname' or '--path pathname'
command-line option.

Alternatively (or in addition), it could get the path from an
environment variable - e.g. if your script is called myscript, then
set and export 'MYSCRIPT_CSV_PATH' in your environment any time before
running it.  You could then have:

 - a hard-coded default
 - which can be overridden by the environment variable
 - which can be overridden by -p or --path on the command-line.


For Bourne-like shells (ash, dash, bash, ksh, etc. even zsh), you have
the choice of either:

 - built in getopts (can only do short single-character options)
 - getopt from util-linux (can do both short and --long options)

I wrote an example back in June, showing/comparing how to use both at:

http://unix.stackexchange.com/a/287344/7696

NOTE: if you use getopt, use ONLY the version from util-linux.  Most
(all?) other versions have serious flaws and are dangerous to use.

For perl, use Getopt::Std for short options, or Getopt::Long for both
short and long.  There's also Getopt::ArgParse, which implements
something a lot like python's argparse in perl.

argparse is probably overkill for your needs but it's worth knowing
about because it's an easy way to implement sub-commands (e.g. like git,
which has numerous subcommands, like 'git add', 'git commit', 'git log',
and many more, each with their own set of options and args)

For python, there's getopt which provides short and long options. if you
need something fancier, use argparse or maybe gflags.

There may be the odd 

Can anyone make suggestions for training / learning resources wrt databases.

[h via luv-main]

Hi,
I am a linux hobbyist.  
I have been using postgis for a while, at a hobby level,
to provide data for web pages and for QGIS.

I am reasonably confident  writing sql statements, and being able to
find the information I need to get the data I am after.

There are two areas I have been unable to find information on:

*  Writing sql / script files
*  Structuring tables either within a database, or on a server

The particular problems I have are:

*  I regularly update my tables from multiple csv files, all residing in
  the same folder. Currently I have a script with a hardwired path for
  each csv file. I would like to have a single 'variable' I could
  change to define the path to all the csvs.

*  I have a database which I have decided should contain three types of
   table  - core definitions / lookups, - raw data and - derived data
   for specific tasks.  I would like to create separate areas in the
   database / (server?) for each table type, or learn of other
   options for how the tables could be structured.


I am NOT looking for specific answers to these problems, but rather
some teaching / groundwork information about relational databases /
sql / postgis concepts.

I have seen training videos / websites covering writing queries and on
database security, but cannot find anything on script files, or
database structure.  I suspect wrong terminology and poor google skills are to 
blame.  

Can anyone suggest web sites etc that provide information on either /
both of these two topics?

With thanks
Hugh
___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

Re: TCP rate/congestion control

[Julien Goodwin via luv-main]

On 16/09/16 18:33, Julien Goodwin via luv-main wrote:
> On 16/09/16 11:02, Toby Corkindale via luv-main wrote:
>> I noticed that Windows 10 now uses CTCP as the default TCP
>> congestion/rate control algorithm, but Linux still defaults to the old
>> Cubic algorithm.
>>
>> CTCP doesn't appear to be available on Ubuntu LTS at the moment, but
>> there's a whole host of others to choose from.
>> Has anyone here worked out which is the best one to use on typical
>> consumer internet links in Australia?
> 
> Over and above the rate control algorithm Linux has a bunch of features
> that make it work much better than a to-the-spec cubic implementation
> (not surprising with a bunch of large content providers like $EMPLOYER
> submitting their fixes upstream).
> 
> Things like TCP pacing and the work from the bufferbloat folk have
> really improved things.
> 
> https://fasterdata.es.net/host-tuning/linux/fair-queuing-scheduler/

...and something that finally went public overnight is a new TCP rate
control algorithm from some of the folk at $EMPLOYER, I for one am very
much looking forward to this hitting upstream and then into a Debian kernel.

https://patchwork.ozlabs.org/patch/671069/



signature.asc
Description: OpenPGP digital signature
___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

Re: OT: trains & demand tariffs [Was: outage]

[Chris Samuel via luv-main]

On Saturday, 17 September 2016 1:23:34 AM AEST Anthony via luv-main wrote:

> This is primarily for "in home displays", but presumably there's gotta be a
> device that's linux friendly out there that talks Zigbee Power v1?

This is what Richard Keech recommended to me:

https://rainforestautomation.com/rfa-z106-raven/

which I bought from these folks:

http://www.fastnetworks.com.au/new-products/9dm7ondccxpc6michdc7182zm0cq7b

and it does work under Linux - at least in so far that it presents a character 
device like a modem to the OS.

The problem is that I live in AusNetServices land where their flash based 
website (sigh) has been saying since 2014 that:

# ZigBee management is currently under pilot and is not currently available.
# Enter access code to continue.

I've prodded them a few times about this and the last time (back in April) 
they told me it might be available in August, but nothing yet. :-(

All the best,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC
___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

Re: pdns security update (was Re: NBN satelite setup)

[Craig Sanders via luv-main]

On Fri, Sep 16, 2016 at 03:27:38PM -0700, Rick Moen wrote:
> > good page that, i've read it before but not for some time. IMO a
> > useful addition to it would be a list of authoritative servers that
> > use bind9 RFC-1034 zonefiles.
>
> You know, they kind of _could_ have called that format the RFC-1034
> file

typo. i actually meant to type 1035 there, and thought i did.

> Anyway, yes, good idea -- and I actually do document RFC 1035 support
> where I know about it.

yep, saw that which is what gave me the idea for a summary list.

> Here's a creative solution from one of the NLnet Labs guys:
> https://www.nlnetlabs.nl/pipermail/nsd-users/2014-August/001998.html

I saw that last night.  It made me realise that probably the best option
for me would be to have NSD listen on 203.16.167.1 while Unbound listens
on 192.168.10.1 (I run both private and public subnets on my LAN so
I can have both private and public hosts and VMs).  Then all I'd have
to do is configure my LAN hosts and VMs to use 192.168.10.1 as the
resolver. Easy.

Unbound seems to have all the features I need, including being able to
forward requests for specific domains to specific servers (useful, e.g.,
for resolving private DNS views over a VPN).

> Other solutions might beckon if the host is multihomed, e.g., bind NSD
> to the public-facing real IP, and bind Unbound to the private RFC1918
> address.

err, yes. exactly that.


> I'm tempted to react 'Fine, let us know when you're done playing
> standards gods, and I'll start paying attention.'

I mostly just leave things alone and then every 2 or 5 years or so go on
a binge of updating everything to the latest standards.

Unless I'm bored, or have a particular reason to make changes.

craig

--
craig sanders 
___
luv-main mailing list
luv-main@luv.asn.au
https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main