[
https://issues.apache.org/jira/browse/BEAM-10812?focusedWorklogId=475984&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-475984
]
ASF GitHub Bot logged work on BEAM-10812:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Aug/20 20:59
Start Date: 28/Aug/20 20:59
Worklog Time Spent: 10m
Work Description: ceocoder commented on a change in pull request #12683:
URL: https://github.com/apache/beam/pull/12683#discussion_r479531914
##########
File path: sdks/go/pkg/beam/io/textio/textio.go
##########
@@ -79,6 +81,7 @@ func expandFn(ctx context.Context, glob string, emit
func(string)) error {
}
func readFn(ctx context.Context, filename string, emit func(string)) error {
+ fmt.Println(filename)
Review comment:
yikes. yes - this was my bad.
##########
File path: sdks/go/pkg/beam/io/textio/textio.go
##########
@@ -93,11 +96,18 @@ func readFn(ctx context.Context, filename string, emit
func(string)) error {
}
defer fd.Close()
- scanner := bufio.NewScanner(fd)
- for scanner.Scan() {
- emit(scanner.Text())
+ rd := bufio.NewReader(fd)
+ for {
+ line, err := rd.ReadString('\n')
+ if err == io.EOF {
Review comment:
ack. done.
##########
File path: sdks/go/pkg/beam/io/textio/textio.go
##########
@@ -93,11 +96,18 @@ func readFn(ctx context.Context, filename string, emit
func(string)) error {
}
defer fd.Close()
- scanner := bufio.NewScanner(fd)
- for scanner.Scan() {
- emit(scanner.Text())
+ rd := bufio.NewReader(fd)
+ for {
+ line, err := rd.ReadString('\n')
Review comment:
ack. done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 475984)
Time Spent: 1h 10m (was: 1h)
> Go textio.Read fails on long lines due to bufio.Scanner limitations
> -------------------------------------------------------------------
>
> Key: BEAM-10812
> URL: https://issues.apache.org/jira/browse/BEAM-10812
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Affects Versions: 2.23.0
> Reporter: Dhaivat Pandit
> Priority: P2
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> bufio.Scanner has an limitations on length of line, it is recommended to use
> [bufio.Reader|https://github.com/golang/go/blob/master/src/bufio/scan.go#L27]
> for arbitrary line lengths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
