Re: [go-nuts] Elementary Question About Calling Functions in Different Packages

[Kurtis Rader]

On Thu, Jun 9, 2022 at 7:06 PM jlfo...@berkeley.edu 
wrote:

> How can this program be made to work? I've tried many things, which all
>>> lead back
>>> to the situation I describe above.
>>>
>>
>> You need to refactor the code to eliminate the import cycle.
>> https://jogendra.dev/import-cycles-in-golang-and-how-to-deal-with-them
>>
>
> As a learning exercise I'm converting a large app into Go.
>

That's going to be a painful way to learn Go. Worse, doing a
straightforward, mechanical, translation of a program written in another
language is likely to result in non-idiomatic Go code and reinforce bad
habits (bad in the context of Go, not the other language).


> The app's source code
> is located in many directories, and I've already learned that all files in
> a package have to be located
> in the same directory so that rules out the suggestion in the posting
> above to put everything in
> one package. (What's the reason for this requirement)?
>

Technically, that is an implementation detail of a particular Go toolchain.
>From https://go.dev/ref/spec#Packages:

> An implementation may require that all source files for a package inhabit
the same directory.

I don't know why the official (primary?) Go toolchain imposes that
restriction. If you google "golang package structure multiple directories"
you can find discussions of this question. Such as
https://stackoverflow.com/questions/45899203/can-i-develop-a-go-package-in-multiple-source-directories
.

I'm guessing the app you're translating is written in Java and you're
comfortable with that language. I've only written a trivial amount of Java
code but did start writing Go a few years ago after using Python as my
primary language for several years (and, C, COBOL, FORTRAN, and many other
languages before that). All I can say is that you really need to avoid
imposing idioms from language J on language G. At least if you want to be
successful at writing high quality code in language G.

So, I'll probably have to keep using multiple packages if I can somehow
> figure out how to solve
> the package import cycle problem.
>

The most common solution is to factor out the aspects which cause the cycle
into a third package that one or both of the current packages imports. In
my four decades of programming I've only seen one import cycle (in Python)
that was hard to remove. My team eventually used the "Python lazy import"
hack. Refactoring the code to eliminate the import cycle, and therefore the
hack to handle it, was high on my team's priority list when I left the
project.

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD9ugevVN3g8%3D%3DfQaCm8vAj4oTfF4tYR6zQf9Yt%3D9UTnJw%40mail.gmail.com.

Re: [go-nuts] Elementary Question About Calling Functions in Different Packages

[jlfo...@berkeley.edu]

Thanks for the quick inciteful response.


> From https://go.dev/ref/spec#Import_declarations:
>
> > It is illegal for a package to import itself, directly or indirectly, 
> or to directly import a package without referring to any of its exported 
> identifiers.
>  
>
I should have researched this. My apologies.
 

> How can this program be made to work? I've tried many things, which all 
>> lead back
>> to the situation I describe above.
>>
>
> You need to refactor the code to eliminate the import cycle. 
> https://jogendra.dev/import-cycles-in-golang-and-how-to-deal-with-them
>

As a learning exercise I'm converting a large app into Go. The app's source 
code
is located in many directories, and I've already learned that all files in 
a package have to be located
in the same directory so that rules out the suggestion in the posting above 
to put everything in
one package. (What's the reason for this requirement)?

So, I'll probably have to keep using multiple packages if I can somehow 
figure out how to solve
the package import cycle problem.

Cordially,
Jon Forrest

 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d5a01fd5-38cd-4dc9-81da-7b6c5aba0890n%40googlegroups.com.

Re: [go-nuts] Elementary Question About Calling Functions in Different Packages

[Kurtis Rader]

On Thu, Jun 9, 2022 at 5:18 PM jlfo...@berkeley.edu 
wrote:

> I'm having trouble understanding what should be a trivial issue.
>
> I have the following file structure:
>
> .
> ├── go.mod
> ├── main.go
> ├── p1
> │   └── p1.go
> └── p2
> └── p2.go
>
> The files are small and simple.
>
>  -- go.mod
> module wp
>
> go 1.18
>
> -- main.go
> // main.go
> package main
>
> import (
> "wp/p1"
> "wp/p2"
> )
>
> func main() {
> p1.P()
> p2.P()
> }
>
> -- p1/p1.go
> // p1.go
> package p1
>
> import (
> "fmt"
> "wp/p2"
> )
>
> func P() {
> fmt.Printf("p1\n")
> p2.P()
> }
>
> -- p2.go
> // p2.go
> package p2
>
> import (
> "fmt"
> "wp/p1"
> )
>
> func P() {
> fmt.Printf("p2\n")
> p1.P()
> }
>
>
> In main.go I'm trying to call a function in packages p1 and p2. In package
> p1 p1.go is trying to call a function in package p2, and in package p2
> p2.go is trying to call a function in package p1. Simple, right?
>
> However, running "go build ." in the top level directory says:
>
> package wp
> imports wp/p1
> imports wp/p2
> imports wp/p1: import cycle not allowed
>
> Is this because p1 imports p2, which imports p1, which imports p2, ...?
>

Yes. From https://go.dev/ref/spec#Import_declarations:

> It is illegal for a package to import itself, directly or indirectly, or
to directly import a package without referring to any of its exported
identifiers.


> How can this program be made to work? I've tried many things, which all
> lead back
> to the situation I describe above.
>

You need to refactor the code to eliminate the import cycle. Exactly how
you do that depends on specifics of the situation which you haven't shared.
If you google "golang import cycle not allowed" you'll find many articles
and stackoverflow questions that discuss this. For example:

https://jogendra.dev/import-cycles-in-golang-and-how-to-deal-with-them
https://quoeamaster.medium.com/golang-gotchas-2-the-curse-of-import-cycle-not-allowed-6abfa3523f57

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-thv0c2OED%2BG5DrbuTciAhw468%2B%2BKSgmQS7aog3oK2%3Dw%40mail.gmail.com.

[go-nuts] Elementary Question About Calling Functions in Different Packages

[jlfo...@berkeley.edu]

I'm having trouble understanding what should be a trivial issue.

I have the following file structure:

.
├── go.mod
├── main.go
├── p1
│   └── p1.go
└── p2
└── p2.go

The files are small and simple.

 -- go.mod
module wp

go 1.18

-- main.go
// main.go
package main

import (
"wp/p1"
"wp/p2"
)

func main() {
p1.P()
p2.P()
}

-- p1/p1.go
// p1.go
package p1

import (
"fmt"
"wp/p2"
)

func P() {
fmt.Printf("p1\n")
p2.P()
}

-- p2.go
// p2.go
package p2

import (
"fmt"
"wp/p1"
)

func P() {
fmt.Printf("p2\n")
p1.P()
}


In main.go I'm trying to call a function in packages p1 and p2. In package 
p1 p1.go is trying to call a function in package p2, and in package p2 
p2.go is trying to call a function in package p1. Simple, right?

However, running "go build ." in the top level directory says:

package wp
imports wp/p1
imports wp/p2
imports wp/p1: import cycle not allowed

Is this because p1 imports p2, which imports p1, which imports p2, ...?

How can this program be made to work? I've tried many things, which all 
lead back
to the situation I describe above.

On a separate but similar note, I originally had another file in the top 
level directory that
contained a function I wanted to call from one of the packages. (I know 
this isn't a good idea, but I'm rewriting existing non-Go code into Go). 
This also failed to build for similar
reasons. Researching the issue showed several postings claiming that this 
wasn't 
possible. Is this true?

Cordially,
Jon Forrest


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d399c4c1-e723-494f-a176-08b7c4ac9774n%40googlegroups.com.

Re: [go-nuts] unrecognized import path "code.google.com/p/go.crypto/openpgp"

['Dan Kortschak' via golang-nuts]

On Thu, 2022-06-09 at 14:21 -0700, Peter Sjolin wrote:
> I attempted to use "code.google.com/p/go.crypto/openpgp" in my
> project and got the following error:
> $ go get code.google.com/p/go.crypto/openpgp
> go: unrecognized import path "code.google.com/p/go.crypto/openpgp":
> parse https://code.google.com/p/go.crypto/openpgp?go-get=1: no go-
> import meta tags (meta tag github.com/golang/go did not match import
> path code.google.com/p/go.crypto/openpgp)
>
> Is there an update to this?
>
> Peter

This now lives here golang.org/x/crypto/openpgp (code.google.com closed
down years ago).

Dan

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1039ac47088072ae6a877aabf101d4f7984de35d.camel%40kortschak.io.

[go-nuts] unrecognized import path "code.google.com/p/go.crypto/openpgp"

[Peter Sjolin]

I attempted to use "code.google.com/p/go.crypto/openpgp" in my project and 
got the following error:
$ go get code.google.com/p/go.crypto/openpgp
go: unrecognized import path "code.google.com/p/go.crypto/openpgp": parse 
https://code.google.com/p/go.crypto/openpgp?go-get=1: no go-import meta 
tags (meta tag github.com/golang/go did not match import path 
code.google.com/p/go.crypto/openpgp)

Is there an update to this?

Peter


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e8faf34c-8309-461d-9399-4d82357cb24an%40googlegroups.com.

[go-nuts] Re: Implementation for Copy-On-Write

[Brian Candler]

You can trim most of it out. Instead of the 'insert' func, just m.Store() 
the new value.

But that isn't Copy-On-Write.  This is simply replacing the value in a 
variable with another value.  Anybody who read out the old value, still has 
their copy of the old value, probably stored in some variable of their 
own.  So really this is copy-on-read, not copy-on-write.

In any case, string and int64 are both immutable types, so you don't have 
to worry whether it's a copy or not.

Note that gotip has extra types like atomic.Int64, to make it easier to 
work with such values (and safer, because there's no way to access them 
accidentally without going through the synchronization interface). 
See: https://pkg.go.dev/sync/atomic@master

On Thursday, 9 June 2022 at 17:41:18 UTC+1 ag9920 wrote:

> Copy-On-Write is a classic way to deal with concurrency problems in the 
> "frequently read, but infrequently updated" scenario.
>
> I noticed an interesting discussion from a previous conversation: 
> https://groups.google.com/g/golang-nuts/c/zyQnord8hyc. The author of 
> weed-fs tried to implement Copy-On-Write by simply assigning a new value to 
> a string variable in a struct. Like this: `vs.masterNode = master`.
>
> So here comes the question, what's the recommend way of implmentating COW?
>
> The atomic package provides an example: 
> https://pkg.go.dev/sync/atomic#example-Value-ReadMostly
>
> import (
>   "sync"
>   "sync/atomic"
> )
>
> func main() {
>   type Map map[string]string
>   var m atomic.Value
>   m.Store(make(Map))
>   var mu sync.Mutex // used only by writers
>   // read function can be used to read the data without further 
> synchronization
>   read := func(key string) (val string) {
> m1 := m.Load().(Map)
> return m1[key]
>   }
>   // insert function can be used to update the data without further 
> synchronization
>   insert := func(key, val string) {
> mu.Lock() // synchronize with other potential writers
> defer mu.Unlock()
> m1 := m.Load().(Map) // load current value of the data structure
> m2 := make(Map)  // create a new value
> for k, v := range m1 {
>   m2[k] = v // copy all data from the current object to the new one
> }
> m2[key] = val // do the update that we need
> m.Store(m2)   // atomically replace the current object with the new one
> // At this point all new readers start working with the new version.
> // The old version will be garbage collected once the existing readers
> // (if any) are done with it.
>   }
>   _, _ = read, insert
> }
>
>
> Acutally, most of the usecase I saw use map as the underlying data 
> structure for demonstration.
>
> What's the correct way to implement COW on a string variable, or maybe 
> other primitive type like int64 ? Should I just replace the `Map` type, and 
> keep everything else the same in the above official example ?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2cf6d4bd-ad5f-4dff-9e2a-21585b98aca4n%40googlegroups.com.

Re: [go-nuts] encoding/asn1 overly strict?

[Brian Candler]

I forgot to add one thing, and you didn't paste the whole certificate PEM 
so I can't check this.

Recent versions of Go won't verify the certificate unless it contains a 
subjectAltName; matching against only the CN is no longer supported.  So if 
you do get your vendor to re-issue the cert, make sure it also includes a 
DNS SAN, if it doesn't already.  It doesn't matter if it's just a string 
like "EEBUS"; you can specify the ServerName at connection time.

On Thursday, 9 June 2022 at 17:38:04 UTC+1 Brian Candler wrote:

> On Thursday, 9 June 2022 at 14:27:04 UTC+1 cpu...@gmail.com wrote:
>
>> We receive an alert 40 (Handshake failure ) when using openssl. So the 
>> cert is definitively faulty in some way. 
>>
>>  :~/wallbox/hack$ openssl s_client  -connect 192.168.1.180:4712 
>>
> Certainly it will report a failed verification, but s_client should 
> continue (i.e. allow you to send and receive data), if the TLS connection 
> was established.  That is: it won't drop the connection on the floor unless 
> you've specified "-verify_return_error"
>
> Have you tried adding "-servername EEBUS" and/or "-verify 0" to the 
> command line?  And/or "-CAFile " where this file contains 
> a copy of the (self-signed) certificate itself.
>
>
> Seems that in this case- if we regard openssl as "the standard" it's 
>> obsolete to talk about Go.
>>
>
> Go doesn't use openssl internally.  I was only trying to establish whether 
> this vendor has *any* claim of their implementation interoperating with any 
> sort of TLS client.  If openssl won't talk to it, then you can pretty much 
> argue they're not talking TLS at all.
>
> However if openssl s_client does talk to it, then you have another 
> workaround: you could exec.Cmd  an instance 
> of openssl s_client, and pipe data to and from it.
>
> But really I would question the wisdom of dealing with any vendor who (a) 
> doesn't follow standards for certificates, and (b) will not allow you to 
> replace the device certificate, or won't replace it themselves, when given 
> direct evidence of its brokenness and non-compliance to standards.  How 
> exactly do you think this vendor will respond when you find a more serious 
> security violation, or some other failing in their product?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f8f1b1ca-c2a5-40e8-b4a0-c5733b06da2fn%40googlegroups.com.

[go-nuts] Implementation for Copy-On-Write

[ag9920]

Copy-On-Write is a classic way to deal with concurrency problems in the 
"frequently read, but infrequently updated" scenario.

I noticed an interesting discussion from a previous conversation: 
https://groups.google.com/g/golang-nuts/c/zyQnord8hyc. The author of 
weed-fs tried to implement Copy-On-Write by simply assigning a new value to 
a string variable in a struct. Like this: `vs.masterNode = master`.

So here comes the question, what's the recommend way of implmentating COW?

The atomic package provides an example: 
https://pkg.go.dev/sync/atomic#example-Value-ReadMostly

import (
  "sync"
  "sync/atomic"
)

func main() {
  type Map map[string]string
  var m atomic.Value
  m.Store(make(Map))
  var mu sync.Mutex // used only by writers
  // read function can be used to read the data without further 
synchronization
  read := func(key string) (val string) {
m1 := m.Load().(Map)
return m1[key]
  }
  // insert function can be used to update the data without further 
synchronization
  insert := func(key, val string) {
mu.Lock() // synchronize with other potential writers
defer mu.Unlock()
m1 := m.Load().(Map) // load current value of the data structure
m2 := make(Map)  // create a new value
for k, v := range m1 {
  m2[k] = v // copy all data from the current object to the new one
}
m2[key] = val // do the update that we need
m.Store(m2)   // atomically replace the current object with the new one
// At this point all new readers start working with the new version.
// The old version will be garbage collected once the existing readers
// (if any) are done with it.
  }
  _, _ = read, insert
}


Acutally, most of the usecase I saw use map as the underlying data 
structure for demonstration.

What's the correct way to implement COW on a string variable, or maybe 
other primitive type like int64 ? Should I just replace the `Map` type, and 
keep everything else the same in the above official example ?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b74eca1a-40c5-4ec8-b025-f48c5037f214n%40googlegroups.com.

Re: [go-nuts] encoding/asn1 overly strict?

[Brian Candler]

On Thursday, 9 June 2022 at 14:27:04 UTC+1 cpu...@gmail.com wrote:

> We receive an alert 40 (Handshake failure ) when using openssl. So the 
> cert is definitively faulty in some way. 
>
>  :~/wallbox/hack$ openssl s_client  -connect 192.168.1.180:4712 
>
Certainly it will report a failed verification, but s_client should 
continue (i.e. allow you to send and receive data), if the TLS connection 
was established.  That is: it won't drop the connection on the floor unless 
you've specified "-verify_return_error"

Have you tried adding "-servername EEBUS" and/or "-verify 0" to the command 
line?  And/or "-CAFile " where this file contains a copy 
of the (self-signed) certificate itself.


Seems that in this case- if we regard openssl as "the standard" it's 
> obsolete to talk about Go.
>

Go doesn't use openssl internally.  I was only trying to establish whether 
this vendor has *any* claim of their implementation interoperating with any 
sort of TLS client.  If openssl won't talk to it, then you can pretty much 
argue they're not talking TLS at all.

However if openssl s_client does talk to it, then you have another 
workaround: you could exec.Cmd  an instance of 
openssl s_client, and pipe data to and from it.

But really I would question the wisdom of dealing with any vendor who (a) 
doesn't follow standards for certificates, and (b) will not allow you to 
replace the device certificate, or won't replace it themselves, when given 
direct evidence of its brokenness and non-compliance to standards.  How 
exactly do you think this vendor will respond when you find a more serious 
security violation, or some other failing in their product?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/91fd5d77-a7af-4298-9cc0-7b97a7c4eae2n%40googlegroups.com.

[go-nuts] Re: How to use thrifter.ToJSON() function

[jake...@gmail.com]

>From your first question, it appears that you have never used Go before. I 
would start with the intro tutorial: A Tour of Go . 
Other resources and tutorials on the official documentation page 
 might also be helpful.

I would also note that the thrift-iterator source 
 is not being actively maintained, 
as the last change was 4 years ago. It also does not appear to have a 
go.mod file, which is now more-or less required. So you may need to fork it 
and add one. 

On Wednesday, June 8, 2022 at 2:35:14 PM UTC-4 1vaishna...@gmail.com wrote:

> Hi all!
>
> I'm working on a project in golang, where I have to convert a thrift 
> string into json. I came across thrifter package. thrift-iterator 
> .   
> Please help me write 
> 1) How to import?
> 2) Import external dependencies using bazel
> 3) Also, does thrifter.ToJSON recursively includes other thrift files from 
> the given thrift file.
> like:
> *xyz.thrift file:*
> namespace
> include"abc.thrift"
> -
> -
>
> Will abc will also be included when converting xyz.thrift file into json, 
> while using thrift.ToJSON.
>
> If there exist some another method to convert thrift into json in golang 
> please let me know.
> Please help. Thanks a lot
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9f915102-56cb-4943-9f2c-ff8af7e8df24n%40googlegroups.com.

Re: [go-nuts] encoding/asn1 overly strict?

[cpu...@gmail.com]

On Wednesday, June 8, 2022 at 5:53:50 PM UTC+2 Brian Candler wrote:

> On Wednesday, 8 June 2022 at 10:09:26 UTC+1 andig wrote:
>
>> We've not found an approach for communicating with the device sofar 
>> unless using patched Go stdlib.
>>
>
> Connect via a proxy like stunnel?
>
> Out of interest, does raw "openssl s_client" allow communication with the 
> device?
>

We receive an alert 40 (Handshake failure ) when using openssl. So the cert 
is definitively faulty in some way. 

 :~/wallbox/hack$ openssl s_client  -connect 192.168.1.180:4712 

CONNECTED(0005)

depth=0 CN = EEBUS, O = EVBox Intelligence, C = NL

verify error:num=18:self signed certificate

verify return:1

depth=0 CN = EEBUS, O = EVBox Intelligence, C = NL

verify return:1

140477570593216:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert 
handshake failure:../ssl/record/rec_layer_s3.c:1528:SSL alert number 40

---

Certificate chain

0 s:CN = EEBUS, O = EVBox Intelligence, C = NL

   i:CN = EEBUS, O = EVBox Intelligence, C = NL

---

Server certificate

-BEGIN CERTIFICATE-
MIIBszCCAVmgAwIBAgIBATAKBggqhkjOPQQDAjA6MQ4wDAYDVQQDDAVFRUJVUzEb 

Seems that in this case- if we regard openssl as "the standard" it's 
obsolete to talk about Go.


> It would seem reasonable to me for InsecureSkipVerify to skip certificates 
> without parsing them at all.  It is, after all, insecure by definition.
>

It doesn't do that as it checks for supported ciphers afterwards, so it 
needs to decode the cert first.
 
Cheers,
Andi

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e7270ed8-35bd-428f-8ec8-69e50d48b0b7n%40googlegroups.com.

[go-nuts] Re: Quick poll: Which of these declarative UI trees looks the most legible to you?

[atd...@gmail.com]

Someone did suggest implementing the tree using structs to reduce 
indentation. 
I thought about it. However, there is no inheritance (thankfully!) and 
types are invariant (no covariance/contravariance) in Go so I'm afraid it 
wouldn't compose. 
Unless I'm missing something? 


On Thursday, June 9, 2022 at 5:59:04 AM UTC+2 Henry wrote:

> Alternatively, you may experiment using structs with exported fields. It 
> may be a bit less verbose.
>
> On Thursday, June 9, 2022 at 8:28:32 AM UTC+7 atd...@gmail.com wrote:
>
>> Hey,
>>
>> Yes, they all look kind of similar because they all use function 
>> composition and are variations of the same tree, just not with the same 
>> level of expansion.
>>
>> I've made a new gist that shows more distinct approaches.
>>
>> https://gist.github.com/atdiar/725844d85d4b9835c58f7f834570ab69
>>
>> Thanks for checking, it's really appreciated.
>>
>>
>>
>> On Wednesday, June 8, 2022 at 12:06:53 PM UTC+2 a2800276 wrote:
>>
>>> To be honest: on first glance, they all look identical to me apart from 
>>> the missing AddClass statements in the  last example.
>>> Would it be possible to have some examples with less subtle differences?
>>>
>>> ¯\_(ツ)_/¯ sorry
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/433aa92c-f1e4-4508-bbe7-9996c2d6ce14n%40googlegroups.com.

Re: [go-nuts] Which is the most efficient way to read STDIN lines 100s of MB long and tens of MB info is passed to it

[Const V]

I did implementations and profiling of following:

*BigGrepStrFnd* - Boyer-Moore (grep)

*BigGrepBytes* - Rabin-Karp

*BigGrepStr - *Rabin-Karp

*BigGrepScan *- search with sliding window

Additionally implemented them using concurrency.

Tested on 100 files containing one 100MB line. Searching for 20 character 
string at the end of the line as worst case scenario.

The program is reading each file and loading the 100MB line in the memory 
for analysis. With 100 files I'm getting consistent results.

Here are the results. I needed to know also the memory requirements.

Created two profiles:

go test -run=NONE -bench=cpuprofile

go test -run=NONE -bench=memprofile

Then analyzed with go tool pprof

Computer Mac-mini M1 8 cores, 16 GB

Concurrent versions are performing the search in parallel on 8 slices of 
the 100MB line.

Fastest

*3.04s* -60.47GB -  BigGrepBytes_Concurrent ~2.85 times faster than non 
concurrent.

8.55s - 60.48GB BigGrepBytes

>From non concurrent versions the fastest is BigGrepStrFnd. Note it has 
bigger memory requirements.

*5.44s *- 72.96GB BigGrepStrFnd - Boyer-Moore (grep) ~1.63 times faster 
than BigGrepBytes

3.82s - 72.93GB BigGrepStrFndConcurrent



go test -run=NONE -bench=cpuprofile

goos: darwin

goarch: arm64

pkg: mobiledatabooks.com/biggrep

Benchmark_100MB_End__20charsBigGrepBytes_Concurrent-8  1   
 7653778209 ns/op

Benchmark_100MB_End__20charsBigGrepBytes_-81   
 12326358083 ns/op

Benchmark_100MB_End__20charsBigGrepScan__-81   
 14934611708 ns/op

Benchmark_100MB_End__20charsBigGrepStr___-81   
 13178685417 ns/op

Benchmark_100MB_End__20charsBigGrepStr___Concurrent-8  1   
 8302944958 ns/op

Benchmark_100MB_End__20charsBigGrepStrFnd-81   
 8765527416 ns/op

Benchmark_100MB_End__20charsBigGrepStrFndConcurrent-8  1   
 7361004166 ns/op

Benchmark_100MB_End__20charsBigGrepScan__Concurrent-8  1   
 8413663834 ns/op

PASS

ok mobiledatabooks.com/biggrep81.204s





Type: cpu

Time: Jun 8, 2022 at 10:15pm (PDT)

Duration: 81.11s, Total samples = 89.36s (110.17%)

Active filters:

   focus=Benchmark

   hide=benchm

   show=Benchmark

Showing nodes accounting for 47.88s, 53.58% of 89.36s total

  flat  flat%   sum%cum   cum%

10.37s 11.60% 11.60% 10.37s 11.60%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepScan__

 8.99s 10.06% 21.67%  8.99s 10.06%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepStr___

 8.55s  9.57% 31.23%  8.55s  9.57%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepBytes_

 5.44s  6.09% 37.32%  5.44s  6.09%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepStrFnd

 4.02s  4.50% 41.82%  4.02s  4.50%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepStr___Concurrent

 3.82s  4.27% 46.09%  3.82s  4.27%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepStrFndConcurrent

 3.65s  4.08% 50.18%  3.65s  4.08%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepScan__Concurrent

 3.04s  3.40% 53.58%  3.04s  3.40%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepBytes_Concurrent







Type: alloc_space

Time: Jun 8, 2022 at 10:17pm (PDT)

Active filters:

   focus=Benchmark

   hide=benchm

   show=Benchmark

Showing nodes accounting for 533.60GB, 100% of 533.61GB total

  flat  flat%   sum%cum   cum%

   72.96GB 13.67% 13.67%72.96GB 13.67%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepStrFnd

   72.95GB 13.67% 27.34%72.95GB 13.67%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepStr___Concurrent

   72.94GB 13.67% 41.01%72.94GB 13.67%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepStr___

   72.93GB 13.67% 54.68%72.93GB 13.67%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepStrFndConcurrent

   60.48GB 11.33% 66.01%60.48GB 11.33%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepBytes_

   60.47GB 11.33% 77.35%60.47GB 11.33%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepBytes_Concurrent

   60.45GB 11.33% 88.67%60.45GB 11.33%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepScan__Concurrent

   60.43GB 11.33%   100%60.43GB 11.33%  
mobiledatabooks.com/biggrep.Benchmark_100MB_End__20charsBigGrepScan__


3.82s
On Sunday, May 8, 2022 at 9:36:34 PM UTC-7 Const V wrote:

> The program is working fine with STDOUT and there is no need of  program > 
> /dev/null.
>
> I was using the Pipe for the unit tests with _test.go to catch the STDOUT.
> How you will write your function as you mention it?
>
> On Sunday, May 8, 2022 at 9:05:51 PM UTC-7 Amnon wrote:
>
>> So what happens when you run your program > /dev/null
>> ?
>>
>> For testing I