Hi all, I've hit a problem that feels like it has a straight-forward answer.
I have a large XML file that I'd like to split up into subfiles of roughly
equal size. My first pass looks like:
import qualified Data.Text as T
import Pipes
import qualified Pipes.Prelude as P
import qualified Pipes.Prelude.Text as PT
import Pipes.Safe
import Text.Printf.TH
-- | Streams lines from the source file.
-- This drops the first three lines, which are not Elements.
xml :: MonadSafe m => FilePath -> Producer T.Text m ()
xml fp = PT.readFileLn fp >-> P.drop 3
-- | Stream an entire OSM Element with all its children (tags, etc).
-- Note: OSM XML is rather flat - children never have children.
element :: Monad m => Pipe T.Text [T.Text] m ()
element = undefined -- `await` lines until you find a closing tag, then
pack as a List and `yield`?
-- | Writes one legal @<osm> ... </osm>@ block.
osm :: Int -> Consumer [T.Text] (SafeT IO) ()
osm !fpn = do
let fp = [s|catalog/out-%d.osm|] fpn
P.take 1000 >-> P.concat >-> PT.writeFileLn fp
osm $ fpn + 1
splitAll :: Effect (SafeT IO) ()
splitAll = xml "somefile" >-> element >-> osm 0
My intent is to stream groups of 1000 XML elements (and their children) to
separate files. Luckily the XML in question is only ever one layer deep,
like:
<foo>
<bar/>
<bar/>
<bar/>
</foo>
So this `foo` group here would count as 1 written element, not 5.
What stands out right away is the type signature of `element`. The
`[T.Text]` feels very unidiomatic, but I couldn't think of another way to
group all the parent and child nodes together in such a way that `osm`
would know it had processed 1000 of such groups.
I read the `pipes-group` tutorial, but it wasn't immediately clear to me if
that's what I needed. I *do* know that at maximum any given `<foo>` parent
can only have a few hundred `<bar>` children, but that still breaks the
output streaming as I wait for the List to populate.
Question: How can I structure things such that `osm` knows when to start
writing to a new file?
Thanks
PS. `osm` defined as it currently is probably also ends up in an infinite
loop.
--
You received this message because you are subscribed to the Google Groups
"Haskell Pipes" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to haskell-pipes+unsubscr...@googlegroups.com.
To post to this group, send email to haskell-pipes@googlegroups.com.
