[
https://issues.apache.org/jira/browse/BEAM-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16407415#comment-16407415
]
Jacob Marble commented on BEAM-3032:
------------------------------------
[~jbonofre] thanks for your offer, I would love your input on this one!
The current branch is working, here's an example of usage:
[https://github.com/jacobmarble/beam-redshift-example]
The unit tests aren't even close; can we discuss the interface first? Some
notes:
* DataSourceConfiguration exists because Redshift's DataSource is not
Serializable
* Redshift.Write wraps Copy; Copy can also be used directly
* Redshift.Read wraps Unload; Unload can also be used directly
* COPY (SQL command) can ingest from S3 in any region; UNLOAD only writes to
S3 in the same region; this implementation does not support the extra feature
in COPY
* RedshiftMarshaller is just a way to convert query results into objects,
happy to improve this
* StringToListAccumulator makes FileSystems.delete() more efficient by helping
the FileSystem implementation to batch delete operations
* Redshift.longestCommonPrefix() is probably over-the-top, but would like a
second opinion before I refactor
* TextIO.Read is instantiated with TextIO.read(); is this pattern preferred
here?
* Same question for .Write
* Copy and Unload lack several available features, but this implementation
probably satisfies 90%+ of potential use
[https://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html]
[https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html]
> Add RedshiftIO
> --------------
>
> Key: BEAM-3032
> URL: https://issues.apache.org/jira/browse/BEAM-3032
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
> Environment: AWS Redshift
> Reporter: Jacob Marble
> Assignee: Jacob Marble
> Priority: Minor
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> I would like to add a RedshiftIO Java extension to perform bulk read/write
> to/from AWS Redshift via the UNLOAD and COPY Redshift SQL commands. This
> requires S3, which is the subject of BEAM-2500.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
