[go-nuts] MarshalJSON() output encrypted json
https://play.golang.org/p/6qKkJhVsnU1
type Payload struct{}
func (p Payload) MarshalJSON() ([]byte, error) {
return []byte("my custom encryption stuf"), nil
}
func main() {
p := Payload{}
j, err := json.Marshal(p)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(j))
}
json: error calling MarshalJSON for type main.Payload: invalid character 'm'
looking for beginning of value
How do you skip the check from json.Marshal and just return the []byte directly
whatever they are?
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/554c8ca8-2bca-4321-9ee5-0524e28422a3%40googlegroups.com.
Re: [go-nuts] Simple worker pool in golnag
I think it is more complex - or simpler :) - than that. A lot depends on the
Kafka client - for example the sarama client recommends one client per
producer/consumer, other clients may multiplex on the same client so having
more than one consumer (sender) may not be beneficial if the IO is fully async
acknowledged.
In general, you want to parallelize (add go routines) to the the portions that
can be parallized (either because they benefit from additional cpu,
scatter/gather IO (network requests or disk), or have independent/multiple
destination output stages) - but you have to pay special attention to any
“ordering” of events that may be required on the consumer side, and the
acknowledgements required on the Kafka side.
In your example, you are still creating 7xNCPU senders, and only 2 producers -
which would mean that each send is completely independent and is a minimum 25x
slower than the producing (given 8 cores x 7 / 2 producers). This could be the
case, but seems unlikely.
> On Dec 26, 2019, at 11:51 PM, Amarjeet Anand
> wrote:
>
> Hi Robert
>
> Actually the code above is simplified to make it easy to understand.
>
> Thanks for the suggestion on variable namings... Will improve that.
>
> The scenario is like the producer functions(produceTaskOfType1ToChan() and
> produceTaskOfType2ToChan()) will produce a list of strings to the channel...
> like...
>
> func produceTaskOfType2ToChan(wg *sync.WaitGroup, autoCancelChan chan string)
> {
> defer wg.Done()
> autoCancelIds := getAutoCancelIdsFromSource2()
> for autoCancelId := range autoCancelIds {
> autoCancelChan <- autoCancelId
> }
> }
>
> Now does this code makes some sense?
>
>
>
> On Fri, 27 Dec, 2019, 10:10 AM robert engels, > wrote:
> Yes, the code doesn’t work :) - it will only ever produce 2 items - unless
> that was expected - even so, you want the N workers doing work, and probably
> a constant number sending to Kafka - but a lot depends on your “serial
> needs”. In your case you only have 2 workers producing work, and N senders -
> which is backwards to me.
>
> I would also say that your variable names could be improved - as
> “autoCancelChan” isn’t really meaningful here, it is simple a chan used to
> send items to the Kafka senders (at least I think).
>
>> On Dec 26, 2019, at 10:12 PM, Amarjeet Anand > > wrote:
>>
>> Hi
>>
>> I have to produce some task to kafka parallely. So I want to implement a
>> simple worker group pattern in go.
>>
>> Does the below code decent enough to take it to production?
>>
>>
>> var workerCount = runtime.NumCPU()*7 + 1
>>
>> func WorkerPattern() {
>> taskWg := {}
>> taskWg.Add(2)
>>
>> autoCancelChan := make(chan string, workerCount*3) // *3, just to make
>> enough room. workers will be slower anyways
>> go produceTaskOfType1ToChan(taskWg, autoCancelChan)
>> go produceTaskOfType2ToChan(taskWg, autoCancelChan)
>>
>> // start workers to push autoCancel to kafka
>> workerWg := {}
>> go kafkaProducerWorkers(autoCancelChan, workerWg)
>>
>> // wait to close autoCancelChan channel till all the task is written
>> taskWg.Wait()
>> close(autoCancelChan)
>>
>> // wait till all workers finish their task
>> workerWg.Wait()
>>
>> fmt.Println("Done!!!")
>> }
>>
>> func produceTaskOfType1ToChan(wg *sync.WaitGroup, autoCancelChan chan
>> string) {
>> defer wg.Done()
>> // can produce random number of task on autoCancelChan
>> autoCancelChan <- "task of type of 1"
>> }
>>
>> func produceTaskOfType2ToChan(wg *sync.WaitGroup, autoCancelChan chan
>> string) {
>> defer wg.Done()
>> // can produce random number of task on autoCancelChan
>> autoCancelChan <- "task of type of 2"
>> }
>>
>> func kafkaProducerWorkers(autoCancelChan chan string, workerWg
>> *sync.WaitGroup) {
>> workerWg.Add(workerCount)
>> for i := 0; i < workerCount; i++ {
>> go produceToKafka(autoCancelChan, workerWg)
>> }
>> }
>>
>> func produceToKafka(autoCancelChan chan string, workerWg *sync.WaitGroup) {
>> defer workerWg.Done()
>>
>> // for loop will terminate once autoCancelChan is closed
>> for autoCancel := range autoCancelChan {
>> KafkaClient.PublishToKafkaTopic(autoCancel)
>> }
>> }
>> Any improvement you can suggest to this code?
>>
>>
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com
>> .
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/CANFuhy8qBjooo_tB_gT0f%3DTE4DaOFqWL5SWwNghy%2BL-eV82KdA%40mail.gmail.com
>>
>>
Re: [go-nuts] Simple worker pool in golnag
Thanks Ian for the resources. Appreciate it a lot. On Fri, Dec 27, 2019 at 10:57 AM Ian Lance Taylor wrote: > On Thu, Dec 26, 2019 at 8:12 PM Amarjeet Anand > wrote: > > > > I have to produce some task to kafka parallely. So I want to implement a > simple worker group pattern in go. > > Bryan has a good talk on this general topic: > > > https://www.youtube.com/watch?v=5zXAHh5tJqQ=PL2ntRZ1ySWBdatAqf-2_125H4sGzaWngM=23=0s > > https://drive.google.com/file/d/1nPdvhB0PutEJzdCq5ms6UI58dp50fcAN/view > > Ian > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CANFuhy9rdDKs_E5z-VKih%2B%2Bd%2BJU5zVV_9aJigHamgVt1Ssr5fw%40mail.gmail.com.
Re: [go-nuts] Simple worker pool in golnag
Hi Robert
Actually the code above is simplified to make it easy to understand.
Thanks for the suggestion on variable namings... Will improve that.
The scenario is like the producer functions(produceTaskOfType1ToChan() and
produceTaskOfType2ToChan()) will produce a list of strings to the
channel... like...
func produceTaskOfType2ToChan(wg *sync.WaitGroup, autoCancelChan chan string) {
defer wg.Done()
autoCancelIds := getAutoCancelIdsFromSource2()
for autoCancelId := range autoCancelIds {
autoCancelChan <- autoCancelId
}}
Now does this code makes some sense?
On Fri, 27 Dec, 2019, 10:10 AM robert engels, wrote:
> Yes, the code doesn’t work :) - it will only ever produce 2 items - unless
> that was expected - even so, you want the N workers doing work, and
> probably a constant number sending to Kafka - but a lot depends on your
> “serial needs”. In your case you only have 2 workers producing work, and N
> senders - which is backwards to me.
>
> I would also say that your variable names could be improved - as
> “autoCancelChan” isn’t really meaningful here, it is simple a chan used to
> send items to the Kafka senders (at least I think).
>
> On Dec 26, 2019, at 10:12 PM, Amarjeet Anand
> wrote:
>
> Hi
>
> I have to produce some task to kafka parallely. So I want to implement a
> simple worker group pattern in go.
>
> Does the below code decent enough to take it to production?
>
> var workerCount = runtime.NumCPU()*7 + 1
>
> func WorkerPattern() {
> taskWg := {}
> taskWg.Add(2)
>
> autoCancelChan := make(chan string, workerCount*3) // *3, just to make
> enough room. workers will be slower anyways
> go produceTaskOfType1ToChan(taskWg, autoCancelChan)
> go produceTaskOfType2ToChan(taskWg, autoCancelChan)
>
> // start workers to push autoCancel to kafka
> workerWg := {}
> go kafkaProducerWorkers(autoCancelChan, workerWg)
>
> // wait to close autoCancelChan channel till all the task is written
> taskWg.Wait()
> close(autoCancelChan)
>
> // wait till all workers finish their task
> workerWg.Wait()
>
> fmt.Println("Done!!!")}
>
> func produceTaskOfType1ToChan(wg *sync.WaitGroup, autoCancelChan chan string)
> {
> defer wg.Done()
> // can produce random number of task on autoCancelChan
> autoCancelChan <- "task of type of 1"}
>
> func produceTaskOfType2ToChan(wg *sync.WaitGroup, autoCancelChan chan string)
> {
> defer wg.Done()
> // can produce random number of task on autoCancelChan
> autoCancelChan <- "task of type of 2"}
>
> func kafkaProducerWorkers(autoCancelChan chan string, workerWg
> *sync.WaitGroup) {
> workerWg.Add(workerCount)
> for i := 0; i < workerCount; i++ {
> go produceToKafka(autoCancelChan, workerWg)
> }}
>
> func produceToKafka(autoCancelChan chan string, workerWg *sync.WaitGroup) {
> defer workerWg.Done()
>
> // for loop will terminate once autoCancelChan is closed
> for autoCancel := range autoCancelChan {
> KafkaClient.PublishToKafkaTopic(autoCancel)
> }}
>
> Any improvement you can suggest to this code?
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CANFuhy8qBjooo_tB_gT0f%3DTE4DaOFqWL5SWwNghy%2BL-eV82KdA%40mail.gmail.com
>
> .
>
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CANFuhy_AOfx9Vpq67tbwUJQhgfCi6Wmnsfu%2BFJ1yS%3DyyBFptkQ%40mail.gmail.com.
Re: [go-nuts] Simple worker pool in golnag
On Thu, Dec 26, 2019 at 8:12 PM Amarjeet Anand wrote: > > I have to produce some task to kafka parallely. So I want to implement a > simple worker group pattern in go. Bryan has a good talk on this general topic: https://www.youtube.com/watch?v=5zXAHh5tJqQ=PL2ntRZ1ySWBdatAqf-2_125H4sGzaWngM=23=0s https://drive.google.com/file/d/1nPdvhB0PutEJzdCq5ms6UI58dp50fcAN/view Ian -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVc%3Dm9f%3D5kEuRdf9F0cTr6t1baHWzQ_C1ZEOvzbUUtrCQ%40mail.gmail.com.
Re: [go-nuts] Simple worker pool in golnag
Yes, the code doesn’t work :) - it will only ever produce 2 items - unless that
was expected - even so, you want the N workers doing work, and probably a
constant number sending to Kafka - but a lot depends on your “serial needs”. In
your case you only have 2 workers producing work, and N senders - which is
backwards to me.
I would also say that your variable names could be improved - as
“autoCancelChan” isn’t really meaningful here, it is simple a chan used to send
items to the Kafka senders (at least I think).
> On Dec 26, 2019, at 10:12 PM, Amarjeet Anand
> wrote:
>
> Hi
>
> I have to produce some task to kafka parallely. So I want to implement a
> simple worker group pattern in go.
>
> Does the below code decent enough to take it to production?
>
>
> var workerCount = runtime.NumCPU()*7 + 1
>
> func WorkerPattern() {
> taskWg := {}
> taskWg.Add(2)
>
> autoCancelChan := make(chan string, workerCount*3) // *3, just to make
> enough room. workers will be slower anyways
> go produceTaskOfType1ToChan(taskWg, autoCancelChan)
> go produceTaskOfType2ToChan(taskWg, autoCancelChan)
>
> // start workers to push autoCancel to kafka
> workerWg := {}
> go kafkaProducerWorkers(autoCancelChan, workerWg)
>
> // wait to close autoCancelChan channel till all the task is written
> taskWg.Wait()
> close(autoCancelChan)
>
> // wait till all workers finish their task
> workerWg.Wait()
>
> fmt.Println("Done!!!")
> }
>
> func produceTaskOfType1ToChan(wg *sync.WaitGroup, autoCancelChan chan string)
> {
> defer wg.Done()
> // can produce random number of task on autoCancelChan
> autoCancelChan <- "task of type of 1"
> }
>
> func produceTaskOfType2ToChan(wg *sync.WaitGroup, autoCancelChan chan string)
> {
> defer wg.Done()
> // can produce random number of task on autoCancelChan
> autoCancelChan <- "task of type of 2"
> }
>
> func kafkaProducerWorkers(autoCancelChan chan string, workerWg
> *sync.WaitGroup) {
> workerWg.Add(workerCount)
> for i := 0; i < workerCount; i++ {
> go produceToKafka(autoCancelChan, workerWg)
> }
> }
>
> func produceToKafka(autoCancelChan chan string, workerWg *sync.WaitGroup) {
> defer workerWg.Done()
>
> // for loop will terminate once autoCancelChan is closed
> for autoCancel := range autoCancelChan {
> KafkaClient.PublishToKafkaTopic(autoCancel)
> }
> }
> Any improvement you can suggest to this code?
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com
> .
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CANFuhy8qBjooo_tB_gT0f%3DTE4DaOFqWL5SWwNghy%2BL-eV82KdA%40mail.gmail.com
>
> .
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/4BD6D8C1-FC0F-4B65-B696-146E3B9E4EFA%40ix.netcom.com.
[go-nuts] Simple worker pool in golnag
Hi
I have to produce some task to kafka parallely. So I want to implement a
simple worker group pattern in go.
Does the below code decent enough to take it to production?
var workerCount = runtime.NumCPU()*7 + 1
func WorkerPattern() {
taskWg := {}
taskWg.Add(2)
autoCancelChan := make(chan string, workerCount*3) // *3, just to
make enough room. workers will be slower anyways
go produceTaskOfType1ToChan(taskWg, autoCancelChan)
go produceTaskOfType2ToChan(taskWg, autoCancelChan)
// start workers to push autoCancel to kafka
workerWg := {}
go kafkaProducerWorkers(autoCancelChan, workerWg)
// wait to close autoCancelChan channel till all the task is written
taskWg.Wait()
close(autoCancelChan)
// wait till all workers finish their task
workerWg.Wait()
fmt.Println("Done!!!")}
func produceTaskOfType1ToChan(wg *sync.WaitGroup, autoCancelChan chan string) {
defer wg.Done()
// can produce random number of task on autoCancelChan
autoCancelChan <- "task of type of 1"}
func produceTaskOfType2ToChan(wg *sync.WaitGroup, autoCancelChan chan string) {
defer wg.Done()
// can produce random number of task on autoCancelChan
autoCancelChan <- "task of type of 2"}
func kafkaProducerWorkers(autoCancelChan chan string, workerWg
*sync.WaitGroup) {
workerWg.Add(workerCount)
for i := 0; i < workerCount; i++ {
go produceToKafka(autoCancelChan, workerWg)
}}
func produceToKafka(autoCancelChan chan string, workerWg *sync.WaitGroup) {
defer workerWg.Done()
// for loop will terminate once autoCancelChan is closed
for autoCancel := range autoCancelChan {
KafkaClient.PublishToKafkaTopic(autoCancel)
}}
Any improvement you can suggest to this code?
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CANFuhy8qBjooo_tB_gT0f%3DTE4DaOFqWL5SWwNghy%2BL-eV82KdA%40mail.gmail.com.
[go-nuts] golang library support for cmpv2 (RFC 4210)
Hi, I m new to Go Language. I wanted to know if there are any libraries in Go which provides support for cmpv2(RFC 4210) protocol for certificate request (IR) and certificate renewal(KUR) I searched in https://go.dev/ and https://pkg.go.dev/, but didnt find any. Can you please let me know if such support is available. Regards, Sriram -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4a8bb125-250d-44e5-8f51-e6bda2a5cf0e%40googlegroups.com.
[go-nuts] What happened to the Pholcus crawler?
I had a notification about some issue I commend on in https://github.com/henrylee2cn/pholcus Pholcus was a crawler written in Go, no English docs. I starred it, along with about 6200 others. I assume the guy was bullied by the Chinese government or parts of it, maybe killed, who knows, because his status shows "Cease to programing and cease to live.", because the readme says strange things: Pholcus (source removed) > Disclaimer > > This software is only used for academic research, but because of frequent > occurrences of crawler developers in China related to news related to > violations, the source code of this library has been deleted. At the same > time, academic enthusiasts who have downloaded this software are strongly > recommended to delete the source code as soon as possible. > Solemnly declare: The user must abide by the relevant laws and regulations > of his location. The user shall bear all the consequences caused by > illegal and illegal use! !! This is translated with Google Translate, the original text can be found in the readme. What happened there? When people get bullied or killed over software then something is wrong (I remind you about the truecrypt incident) then a Streisand effect must happen, e.g. a spotlight needs to shine on that issue. Maybe it was removed for copyright violations (the right holders suing the crawlers, might as well happen in the EU with the new laws passed recently and I hear the US is having similar talks). Maybe it's just a marketing ploy, https://github.com/pholcus-go/pholcus exists with a fork from 25 nov 2019, the github organisation was created on 24 dec 2019, the same day the https://github.com/henrylee2cn/pholcus repo was zeroed. If anyone knows what happened, please enlighten us. I find that all very suspicious. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d55b1eaa-ffbd-48ba-8755-43ebbb7f89ae%40googlegroups.com.
[go-nuts] Re: issue with yaml processing
You want to add the flow directive to the struct's attributes. From the
docs:
flow Marshal using a flow style (useful for structs,
sequences and maps).
type Config struct {
Contents []string `yaml:"contents,flow"`
}
Howard
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/65141c7b-c39d-4cd0-af40-d69203e6371d%40googlegroups.com.
[go-nuts] Re: Question regarding a very tricky build set up
On Sunday, 22 December 2019 08:08:56 UTC, christoph...@gmail.com wrote: > > Oh my gosh. You could have saved all the time for writing that long long > question if someone had told you before that GOPATH is a thing of the past. > > Go has a new package dependency management system called Go Modules, and > this removes the need for a central source code workspace. > > See https://github.com/golang/go/wiki/Modules to get started, search > https://blog.golang.org for its Go Module posts, or search the Web for > "Go Modules". The problem is that the documentation on golang.org is still littered with instructions about the GOPATH way of doing things. Happily there is a review in flight to fix this https://go-review.googlesource.com/c/website/+/199417 which hopefully will land soon. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f9e86744-6249-45b4-b979-cc3297c387d9%40googlegroups.com.
