Re: [go-nuts] NTFS deduplicated files changed from regular to symlink in go 1.21
[ + Quin Muntal ] On Thu, Sep 21, 2023 at 10:34 AM Simon Frei wrote: > > Hi, > > We got a report in syncthing that some files fail to sync on an ntfs > filesystem with deduplication enabled after upgrading the app: > https://github.com/syncthing/syncthing/issues/9120 > The only change relevant to filesystem handling there was going from go1.20 > to go1.21. Debug logging shows that `IsRegular` returns false on `LStat` > result of this file, while this is a regular (though possibly deduplicated) > file so should be true and was true before. > > This change in go 1.21 seems like it's related, as it deals with reparse > points: > https://github.com/golang/go/commit/3e44b7d07a7b3c6233eb1bf4cf3cb00a0b85adec > And so does ntfs deduplication according to this doc: > https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/understand > > Now my question is if this was an intentional change, or if this is a > regression and should be filed as a bug. The commit message suggest it might > more likely be a regression, as it seems focused on performance and mentions > symlinks being rare, so I doubt ntfs deduplication was on the radar. > > Any comments about this, specifically if you feel like this might be a > regression/should be filed as a bug. I haven't tried to repro, don't have a > windows ntfs setup handy, but the issue seems clear enough to start reason > about it. > > Cheers, > Simon > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/c7b9a889-b659-4022-a5f1-439bde7e7b8fn%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVciWdNmkizj8bvv5MF4mib9fg%3DUOTeysMjQ18uAxYUNA%40mail.gmail.com.
Re: [go-nuts] Go 1.21.0 panics on macos ventura 13.5.2
On Thu, Sep 21, 2023 at 2:16 PM Pierre Laplante wrote: > > Running make planic with: > > fatal error: forcegc: phase error Note for mailing list: this was reported as https://go.dev/issue/63151. Ian -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcU8T%2BqRMPRGDhRn5h8Tkyi_O%3DV33cRS3s%2BynJMmRW3j-A%40mail.gmail.com.
[go-nuts] Go 1.21.0 panics on macos ventura 13.5.2
main.go:
package main
import "plugin"
func main() {
p, err := plugin.Open("../my_plugin/my_plugin.so")
if err != nil {
panic(err)
}
v, err := p.Lookup("V")
if err != nil {
panic(err)
}
f, err := p.Lookup("F")
if err != nil {
panic(err)
}
*v.(*int) = 7
f.(func())() // prints "Hello, number 7"
}
my_plugin
package main
import "fmt"
var V int
func F() { fmt.Printf("Hello, number %d\n", V) }
go version
go version go1.21.1 darwin/arm64
Running make planic with:
fatal error: forcegc: phase error
goroutine 5 [running]:
runtime.throw({0x129d8dc52?, 0x102a4ef10?})
/usr/local/go/src/runtime/panic.go:1077 +0x40 fp=0x1446790
sp=0x1446760 pc=0x129cf29e0
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:319 +0x114 fp=0x14467d0
sp=0x1446790 pc=0x129cf5d64
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1197 +0x4 fp=0x14467d0
sp=0x14467d0 pc=0x129d1f164
created by runtime.init.6 in goroutine 1
/usr/local/go/src/runtime/proc.go:310 +0x24
goroutine 1 [runnable]:
os.NewFile(0x1?, {0x129d8c228?, 0xb?})
/usr/local/go/src/os/file_unix.go:105 +0xb8 fp=0x1400011faa0
sp=0x1400011faa0 pc=0x129d7d1c8
os.init()
/usr/local/go/src/os/file.go:66 +0x310 fp=0x1400011fad0 sp=0x1400011faa0
pc=0x129d7e970
runtime.doInit1(0x129e8fc20)
/usr/local/go/src/runtime/proc.go:6740 +0xd4 fp=0x1400011fc00
sp=0x1400011fad0 pc=0x10298f6f4
runtime.doInit({0x129e8fee0, 0xa, 0x141e0f0?})
/usr/local/go/src/runtime/proc.go:6707 +0x38 fp=0x1400011fc20
sp=0x1400011fc00 pc=0x10298f5d8
plugin.open({0x1029b8f84, 0x19})
/usr/local/go/src/plugin/plugin_dlopen.go:95 +0x520 fp=0x1400011fea0
sp=0x1400011fc20 pc=0x1029b4fe0
plugin.Open(...)
/usr/local/go/src/plugin/plugin.go:80
main.main()
/Users/laplante/go/src/my_main/main.go:6 +0x30 fp=0x1400011ff30
sp=0x1400011fea0 pc=0x1029b5710
runtime.main()
/usr/local/go/src/runtime/proc.go:267 +0x2bc fp=0x1400011ffd0
sp=0x1400011ff30 pc=0x102981a8c
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1197 +0x4 fp=0x1400011ffd0
sp=0x1400011ffd0 pc=0x1029a9ac4
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xc8 fp=0x1446f90
sp=0x1446f70 pc=0x102981eb8
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:322 +0xb8 fp=0x1446fd0
sp=0x1446f90 pc=0x102981d48
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1197 +0x4 fp=0x1446fd0
sp=0x1446fd0 pc=0x1029a9ac4
created by runtime.init.6 in goroutine 1
/usr/local/go/src/runtime/proc.go:310 +0x24
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:398 +0xc8 fp=0x1447760
sp=0x1447740 pc=0x102981eb8
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
/usr/local/go/src/runtime/mgcsweep.go:280 +0xa0 fp=0x14477b0
sp=0x1447760 pc=0x10296dad0
runtime.gcenable.func1()
/usr/local/go/src/runtime/mgc.go:200 +0x28 fp=0x14477d0
sp=0x14477b0 pc=0x102962698
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1197 +0x4 fp=0x14477d0
sp=0x14477d0 pc=0x1029a9ac4
created by runtime.gcenable in goroutine 1
/usr/local/go/src/runtime/mgc.go:200 +0x6c
goroutine 4 [GC scavenge wait]:
runtime.gopark(0x147?, 0x1029d1eb0?, 0x1?, 0x0?, 0x14031e0?)
/usr/local/go/src/runtime/proc.go:398 +0xc8 fp=0x1447f50
sp=0x1447f30 pc=0x102981eb8
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x102a503c0)
/usr/local/go/src/runtime/mgcscavenge.go:425 +0x5c fp=0x1447f80
sp=0x1447f50 pc=0x10296b1dc
runtime.bgscavenge(0x0?)
/usr/local/go/src/runtime/mgcscavenge.go:653 +0x44 fp=0x1447fb0
sp=0x1447f80 pc=0x10296b754
runtime.gcenable.func2()
/usr/local/go/src/runtime/mgc.go:201 +0x28 fp=0x1447fd0
sp=0x1447fb0 pc=0x102962638
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1197 +0x4 fp=0x1447fd0
sp=0x1447fd0 pc=0x1029a9ac4
created by runtime.gcenable in goroutine 1
/usr/local/go/src/runtime/mgc.go:201 +0xac
goroutine 6 [runnable]:
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:176 fp=0x14487d0 sp=0x14487d0
pc=0x129cd4db0
runtime.goexit()
/usr/local/go/src/runtime/asm_arm64.s:1197 +0x4 fp=0x14487d0
sp=0x14487d0 pc=0x129d1f164
created by runtime.createfing in goroutine 1
/usr/local/go/src/runtime/mfinal.go:163 +0x84
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/c5e8faf6-d65b-44f0-9ba9-a3b54786ed44n%40googlegroups.com.
[go-nuts] Re: golang splitting string by space but considering quoted section as a single part
// SplitString split string with a rune comma ignore quoted
func SplitString(str string, r rune) []string {
quoted := false
return strings.FieldsFunc(str, func(r1 rune) bool {
if r1 == '\'' {
quoted = !quoted
}
return !quoted && r1 == r
})
}
// TestSpitString test case
func TestSpitString(t *testing.T) {
str := " arg1 arg2 'hello world' "
arr := SplitString(str, ' ')
fmt.Println(len(arr))
for i, line := range arr {
fmt.Println(i, "---", line)
}
}
在2019年6月3日星期一 UTC+8 11:35:24 写道：
> You might find http://godoc.org/bitbucket.org/creachadair/shell useful
> also.
>
> –M
>
>
> On Saturday, June 1, 2019 at 8:14:14 PM UTC-7, AJ ONeal wrote:
>>
>> Based on the work of others here I created my own version that passes the
>> tests I needed to pass:
>>
>> * strips outer quotes
>> * keeps inner quotes
>> * empty quotes produce empty string
>>
>> https://play.golang.org/p/NzYUsZ-pm2v
>>
>> const NullStr = rune(0)
>>
>>
>>
>> // ParseArgs will parse a string that contains quoted strings the same as
>>> bash does
>>> // (same as most other *nix shells do). This is secure in the sense that
>>> it doesn't do any
>>> // executing or interpeting. However, it also doesn't do any escaping,
>>> so you shouldn't pass
>>> // these strings to shells without escaping them.
>>> func ParseArgs(str string) ([]string, error) {
>>> var m []string
>>> var s string
>>
>>
>>
>> str = strings.TrimSpace(str) + " "
>>> lastQuote := NullStr
>>> isSpace := false
>>
>>
>>
>> for i, c := range str {
>>> switch {
>>> // If we're ending a quote, break out and skip this character
>>> case c == lastQuote:
>>> lastQuote = NullStr
>>
>>
>>
>> // If we're in a quote, count this character
>>> case lastQuote != NullStr:
>>> s += string(c)
>>
>>
>>
>> // If we encounter a quote, enter it and skip this character
>>> case unicode.In(c, unicode.Quotation_Mark):
>>> isSpace = false
>>> lastQuote = c
>>
>>
>>
>> // If it's a space, store the string
>>> case unicode.IsSpace(c):
>>> if 0 == i || isSpace {
>>> continue
>>> }
>>> isSpace = true
>>> m = append(m, s)
>>> s = ""
>>
>>
>>
>> default:
>>> isSpace = false
>>> s += string(c)
>>> }
>>> }
>>> if lastQuote != NullStr {
>>> return nil, fmt.Errorf("Quotes did not terminate")
>>> }
>>> return m, nil
>>> }
>>
>>
>> I may or may not keep this updated at
>> https://git.coolaj86.com/coolaj86/git-scripts/src/branch/master/git-proxy
>>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/797da957-897c-476d-960a-9c02f5fd6a70n%40googlegroups.com.
[go-nuts] NTFS deduplicated files changed from regular to symlink in go 1.21
Hi, We got a report in syncthing that some files fail to sync on an ntfs filesystem with deduplication enabled after upgrading the app: https://github.com/syncthing/syncthing/issues/9120 The only change relevant to filesystem handling there was going from go1.20 to go1.21. Debug logging shows that `IsRegular` returns false on `LStat` result of this file, while this is a regular (though possibly deduplicated) file so should be true and was true before. This change in go 1.21 seems like it's related, as it deals with reparse points: https://github.com/golang/go/commit/3e44b7d07a7b3c6233eb1bf4cf3cb00a0b85adec And so does ntfs deduplication according to this doc: https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/understand Now my question is if this was an intentional change, or if this is a regression and should be filed as a bug. The commit message suggest it might more likely be a regression, as it seems focused on performance and mentions symlinks being rare, so I doubt ntfs deduplication was on the radar. Any comments about this, specifically if you feel like this might be a regression/should be filed as a bug. I haven't tried to repro, don't have a windows ntfs setup handy, but the issue seems clear enough to start reason about it. Cheers, Simon -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c7b9a889-b659-4022-a5f1-439bde7e7b8fn%40googlegroups.com.
Re: [go-nuts] Encrypting a small secret using curve25519
You can use CMS encryption (part of https://datatracker.ietf.org/doc/html/rfc5652) For example you can try https://github.com/mozilla-services/pkcs7 In CMS you use a public key to encrypt the generated intermediate symmetric key (content encryption key) and the receiver can decrypt this key and then decrypt content with it. On Wed, Sep 20, 2023 at 10:02 AM christoph...@gmail.com wrote: > > Hello, > > I noticed that the go standard library only support ed25519 signing > (https://pkg.go.dev/crypto/ed25519@go1.21.1). > > I would need to encrypt a small secret with the public key of the receiver so > that he is the only one able to decrypt it with its private key. The small > secret would typically be a random symmetric key used to encrypt the possibly > long message. > > The only solution I found is to use nacl.Box > (https://pkg.go.dev/golang.org/x/crypto/nacl/box). Why is it so ? > > Are there alternative reliable go packages I could use ? I'll use only a pure > Go package, not a libsodium wrapper package. > > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/8018a90a-fae2-4c45-8c19-ed8b5c205319n%40googlegroups.com. -- Kind regards, Alexey -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMDzThjNmR2KWy16CXPEQ1ezrW0bkK3wVAWk4B7B6yReqXrvqQ%40mail.gmail.com.
[go-nuts] Re: Why doesn't the database/sql package in Go support using placeholders "?" to replace the database name and username in SQL statements?
Thanks for your reply. So does substituting placeholders in the driver still involve pre-compiling the SQL statement to prevent SQL injection? Acutally I am more concerned about security aspects rather than performance overhead. 在2023年9月19日星期二 UTC+8 07:45:35 写道： > Hi. I'm maintainer of go-mysql-driver. > > There is an option to substitute placeholders in the driver, instead of in > the MySQL server. > It can be used to: > > * Reduce roundtrip to execute a query > * Avoid limitations of where placeholders can be used > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/08a09432-87a9-4c97-9a77-65dffaf4b466n%40googlegroups.com.
[go-nuts] Re: Why doesn't the database/sql package in Go support using placeholders "?" to replace the database name and username in SQL statements?
Thanks for your reply. I understand now that the issue does not lie with
the database/sql interface. It appears to be a limitation on the usage of
placeholders in MySQL.
在2023年9月18日星期一 UTC+8 20:27:11 写道：
> Or else it's a prepared statement which gets invoked with parameters.
>
> Mysql's own documentation is unclear on where placeholders can be used:
> https://dev.mysql.com/doc/refman/8.0/en/sql-prepared-statements.html
>
> Note that they give this example:
>
> mysql> SET @table = 't1'; mysql> SET @s = CONCAT('SELECT * FROM ', @table)
> ;
>
> mysql> PREPARE stmt3 FROM @s; mysql> EXECUTE stmt3;
>
> They don't do "SELECT * FROM ?" because you're not allowed to use a
> placeholder for the table name - it must be inlined into the SQL. (And in
> this example @table had better be from a trusted source, since they don't
> do any quoting or escaping)
>
> On Monday, 18 September 2023 at 13:01:52 UTC+1 Vladimir Varankin wrote:
>
>> A thing, that it may be valuable to explain further, is that Go's
>> "database/sql" doesn't come with a built-in query builder.
>>
>> The package implements the database connection pooling/management, but it
>> passes the user's SQL input and its arguments to the "driver". Depending on
>> the particular database kind, the driver may or may not try to interpret
>> the query (and the args), before it passes them to the database server.
>>
>> In the specific example of MySQL/MariaDB, the support for placeholder "?"
>> is a part of this database's flavour of SQL. Thus, it's likely, the driver
>> you're using, passes the query with a placeholder to the DB server, but the
>> DB's support of placeholders is limited to only a subset of queries (or
>> places inside a query) — as the link to SO's answer, shared previously,
>> explained.
>>
>> Hope this makes it a little bit more clear.
>>
>> On Sunday, September 17, 2023 at 10:45:27 AM UTC+2 Brian Candler wrote:
>>
>>> According to this SO answer, CREATE USER in mysql doesn't support
>>> placeholders:
>>>
>>> https://stackoverflow.com/questions/20647115/how-to-write-create-user-with-mysql-prepared-statement
>>> *"+1 for a legitmiate use of QUOTE() which is really the sanest thing
>>> that can be done here, since CREATE USER doesn't support ? placeholders"*
>>>
>>> You can test this by trying a simple DML statement, e.g.
>>> UPDATE users SET name=? where name=?
>>>
>>> On Sunday, 17 September 2023 at 01:02:08 UTC+1 John Zh wrote:
>>>
Hi !
I am going to make some kind of manager app over MySQL clusters by
using Golang. But I found that when I try to exec some SQL line includes
user name or db name, the SQL line can't be correctly parameterized.
For example:
Using GORM based on database/sql or directly using database/sql
```
err := db.Exec("CREATE USER ? IDENTIFIED BY ?", a.Name, a.Pwd).Error
```
Got
```
[1.824ms] [rows:0] CREATE USER 'Reiis' IDENTIFIED BY '12345'
Error 1064 (42000): You have an error in your SQL syntax; check the
manual that corresponds to your MySQL server version for the right syntax
to use near '? IDENTIFIED BY ?' at line 1
```
Seems like it does not replace "?" with a.Name, but rather passes the
SQL command with "?" directly to MySQL. What is more wired, it prints
the SQL command with correctly replaced parameters in the log.
I don't know the the underlying reason behind this phenomenon, is it
intentionally designed like that?
Thx!
>>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/0fdfe954-8297-40e1-a906-17a9d59b56a3n%40googlegroups.com.
[go-nuts] Re: Why doesn't the database/sql package in Go support using placeholders "?" to replace the database name and username in SQL statements?
Thanks for your reply. You have deepened my understanding of placeholders.
It seems to be the limitations of using placeholders in MySQL.
在2023年9月18日星期一 UTC+8 20:01:52 写道：
> A thing, that it may be valuable to explain further, is that Go's
> "database/sql" doesn't come with a built-in query builder.
>
> The package implements the database connection pooling/management, but it
> passes the user's SQL input and its arguments to the "driver". Depending on
> the particular database kind, the driver may or may not try to interpret
> the query (and the args), before it passes them to the database server.
>
> In the specific example of MySQL/MariaDB, the support for placeholder "?"
> is a part of this database's flavour of SQL. Thus, it's likely, the driver
> you're using, passes the query with a placeholder to the DB server, but the
> DB's support of placeholders is limited to only a subset of queries (or
> places inside a query) — as the link to SO's answer, shared previously,
> explained.
>
> Hope this makes it a little bit more clear.
>
> On Sunday, September 17, 2023 at 10:45:27 AM UTC+2 Brian Candler wrote:
>
>> According to this SO answer, CREATE USER in mysql doesn't support
>> placeholders:
>>
>> https://stackoverflow.com/questions/20647115/how-to-write-create-user-with-mysql-prepared-statement
>> *"+1 for a legitmiate use of QUOTE() which is really the sanest thing
>> that can be done here, since CREATE USER doesn't support ? placeholders"*
>>
>> You can test this by trying a simple DML statement, e.g.
>> UPDATE users SET name=? where name=?
>>
>> On Sunday, 17 September 2023 at 01:02:08 UTC+1 John Zh wrote:
>>
>>> Hi !
>>> I am going to make some kind of manager app over MySQL clusters by using
>>> Golang. But I found that when I try to exec some SQL line includes user
>>> name or db name, the SQL line can't be correctly parameterized.
>>> For example:
>>> Using GORM based on database/sql or directly using database/sql
>>> ```
>>> err := db.Exec("CREATE USER ? IDENTIFIED BY ?", a.Name, a.Pwd).Error
>>> ```
>>> Got
>>> ```
>>> [1.824ms] [rows:0] CREATE USER 'Reiis' IDENTIFIED BY '12345'
>>> Error 1064 (42000): You have an error in your SQL syntax; check the
>>> manual that corresponds to your MySQL server version for the right syntax
>>> to use near '? IDENTIFIED BY ?' at line 1
>>> ```
>>>
>>> Seems like it does not replace "?" with a.Name, but rather passes the
>>> SQL command with "?" directly to MySQL. What is more wired, it prints
>>> the SQL command with correctly replaced parameters in the log.
>>>
>>> I don't know the the underlying reason behind this phenomenon, is it
>>> intentionally designed like that?
>>> Thx!
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/b0376b5e-2589-4810-918c-74336e940f30n%40googlegroups.com.
