[PATCH] lzloader miscompiles with gcc 6.2

[Rick Payne]

-O2 optimisation on lzloader when compiled with gcc 6.2.0 causes
the resulting image to fail to boot. Reducing the optimisation
resolves this problem.

Fixes #913

Signed-off-by: Rick Payne 
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 0850f7c..8dd2371 100644
--- a/Makefile
+++ b/Makefile
@@ -458,7 +458,7 @@ $(out)/loader-stripped.elf.lz.o: $(out)/loader-stripped.elf 
$(out)/fastlz/lz
 
 $(out)/fastlz/lzloader.o: fastlz/lzloader.cc | generated-headers
$(makedir)
-   $(call quiet, $(CXX) $(CXXFLAGS) -O2 -m32 -fno-instrument-functions -o 
$@ -c fastlz/lzloader.cc, CXX $<)
+   $(call quiet, $(CXX) $(CXXFLAGS) -O0 -m32 -fno-instrument-functions -o 
$@ -c fastlz/lzloader.cc, CXX $<)
 
 $(out)/lzloader.elf: $(out)/loader-stripped.elf.lz.o $(out)/fastlz/lzloader.o 
arch/x64/lzloader.ld \
$(out)/fastlz/fastlz.o
-- 
2.7.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[COMMIT osv master] aarch64: gcc 7: fix build

[Commit Bot]

From: Nadav Har'El 
Committer: Nadav Har'El 
Branch: master

aarch64: gcc 7: fix build

In commit 05baa614c460f7033828adf926ce0224df40fd8f we fixed code which
wrongly uses "new" on types with too strict requirements. We forgot to
fix one place specific to the aarch64 architecture, and it breaks
compilation of aarch64 on gcc 7.2.1. So let's fix it now.

Signed-off-by: Nadav Har'El 

---
diff --git a/arch/aarch64/smp.cc b/arch/aarch64/smp.cc
--- a/arch/aarch64/smp.cc
+++ b/arch/aarch64/smp.cc
@@ -10,6 +10,7 @@
 #include 
 #include 
 #include 
+#include 
 #include "processor.hh"
 #include "psci.hh"
 #include "arch-dtb.hh"
@@ -62,7 +63,7 @@ void smp_init()
 sched::current_cpu = sched::cpus[0];

 for (auto c : sched::cpus) {
-c->incoming_wakeups = new  
sched::cpu::incoming_wakeup_queue[sched::cpus.size()];
+c->incoming_wakeups =  
aligned_array_new(sched::cpus.size());

 }
 }

--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[COMMIT osv master] scripts/setup.py: support Fedora 27

[Commit Bot]

From: Nadav Har'El 
Committer: Nadav Har'El 
Branch: master

scripts/setup.py: support Fedora 27

No changes appear to be needed in Fedora 27 vs what we needed in Fedora 26.

Signed-off-by: Nadav Har'El 

---
diff --git a/scripts/setup.py b/scripts/setup.py
--- a/scripts/setup.py
+++ b/scripts/setup.py
@@ -91,7 +91,14 @@ class Fedora_26(object):
 ec2_post_install = None
 version = '26'

-versions = [Fedora_19, Fedora_20, Fedora_21, Fedora_22, Fedora_23,  
Fedora_24, Fedora_25, Fedora_26]

+class Fedora_27(object):
+packages =  
['java-1.8.0-openjdk', 'python2-requests', 'compat-openssl10-devel']

+ec2_packages = []
+test_packages = []
+ec2_post_install = None
+version = '27'
+
+versions = [Fedora_19, Fedora_20, Fedora_21, Fedora_22, Fedora_23,  
Fedora_24, Fedora_25, Fedora_26, Fedora_27]


 class RHELbased(Fedora):
 name = ['Scientific Linux', 'NauLinux', 'CentOS Linux',

--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH v3 1/2] Split httpserver module into httpserver-api and httpserver-html5-gui

[Nadav Har'El]

On Mon, Aug 21, 2017 at 10:54 PM, Waldemar Kozaczuk 
wrote:

> Split httpserver into httpserver-api, httpserver-html5-gui and
> httpserver modules. The new httpserver-api module is a subset of original
> httpserver and implements same OSv REST API without any modularity or
> other functionality changes.
> Another new httpserver-html5-gui module contains static content of
> swagger-ui and
> osv-gui HTML5 apps and depends on httpserver-api. Remaining httpserver
> module provides
> backwards-compatibility and combines httpserver-api and
> httpserver-html5-gui functionality.
>
> Updated makefiles in httpserver-jolokia-plugin, httpserver-jvm-plugin and
> cloud-init to
> point to new locations of header files.
>
> Fixed #822
>
>
I discovered a weird problem caused (in a roundabout way) by this patch.

I upgraded my build machine to Fedora 27, and "scripts/build" (the default,
CLI, image) failed when it tries to upload a
"/lib64/libboost_system.so.1.63.0" which doesn't exist (Fedora 27 has
1.64.0).
Where does this file get into the manifest?

Turns out I had a very old modules/httpserver/usr.manifest which contained
this path...
This usr.manifest was generated by httpserver's Makefile before the split,
but after the split it just remained there.
Since there is no more Makefile in httpserver, even a "scripts/build clean"
(which I tried...) doesn't remove this old file.

Should we perhaps add a Makefile with a "clean" target to
modules/httpserver just to remove this old file?

Of course this problem will not affter users who checked out OSv from
scratch. It only affects people (like me) who are using the same working
directory for a very long time.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[COMMIT osv master] scripts/build: clearer message and no hang in case of missing file

[Commit Bot]

From: Nadav Har'El 
Committer: Nadav Har'El 
Branch: master

scripts/build: clearer message and no hang in case of missing file

Before this patch, if one of the files in the manifest is missing,
scripts/build prints the problem (as a long python stack trace) but this
message is mixed with the guest's output (of which files were saved in the
image), and the guest does not exit. This looks to the user like a silent
"hang" in the build (the error message is often not even visible - the user
has to scroll up to see it).

With this patch, when we have a problem uploading a file, we stop printing
the guest's output, print a clear message about the failure and stop the
guest. scripts/build will then say the build failed, and the reason will
be clear. And no hang.

Fixes #925

Signed-off-by: Nadav Har'El 

---
diff --git a/scripts/upload_manifest.py b/scripts/upload_manifest.py
--- a/scripts/upload_manifest.py
+++ b/scripts/upload_manifest.py
@@ -30,10 +30,12 @@ def upload(osv, manifest, depends):
 # We'll want to read the rest of the guest's output, so that it doesn't
 # hang, and so the user can see what's happening. Easiest to do this  
with

 # a thread.
-def consumeoutput(file):
+stop_output = False
+def consumeoutput(file, silent):
 for line in iter(lambda: file.readline(), b''):
-os.write(sys.stdout.fileno(), line)
-threading.Thread(target=consumeoutput, args=(osv.stdout,)).start()
+if not silent():
+os.write(sys.stdout.fileno(), line)
+threading.Thread(target=consumeoutput, args=(osv.stdout, lambda:  
stop_output)).start()


 # Send a CPIO header or file, padded to multiple of 4 bytes
 def cpio_send(data):
@@ -63,35 +65,51 @@ def cpio_header(filename, mode, filesize):
 + filename + b'\0')

 # Send the files to the guest
-for name, hostname in files:
-if hostname.startswith("->"):
-link = hostname[2:]
-cpio_send(cpio_header(name, stat.S_IFLNK, len(link)))
-cpio_send(link.encode())
-else:
-depends.write('\t%s \\\n' % (hostname,))
-if hostname.endswith("-stripped.so"):
-continue
-hostname = strip_file(hostname)
-if os.path.islink(hostname):
-perm = os.lstat(hostname).st_mode & 0o777
-link = os.readlink(hostname)
-cpio_send(cpio_header(name, perm | stat.S_IFLNK,  
len(link)))

+try:
+for name, hostname in files:
+if hostname.startswith("->"):
+link = hostname[2:]
+cpio_send(cpio_header(name, stat.S_IFLNK, len(link)))
 cpio_send(link.encode())
-elif os.path.isdir(hostname):
-perm = os.stat(hostname).st_mode & 0o777
-cpio_send(cpio_header(name, perm | stat.S_IFDIR, 0))
 else:
-perm = os.stat(hostname).st_mode & 0o777
-cpio_send(cpio_header(name, perm | stat.S_IFREG,  
os.stat(hostname).st_size))

-with open(hostname, 'rb') as f:
-cpio_send(f.read())
+depends.write('\t%s \\\n' % (hostname,))
+if hostname.endswith("-stripped.so"):
+continue
+hostname = strip_file(hostname)
+if os.path.islink(hostname):
+perm = os.lstat(hostname).st_mode & 0o777
+link = os.readlink(hostname)
+cpio_send(cpio_header(name, perm | stat.S_IFLNK,  
len(link)))

+cpio_send(link.encode())
+elif os.path.isdir(hostname):
+perm = os.stat(hostname).st_mode & 0o777
+cpio_send(cpio_header(name, perm | stat.S_IFDIR, 0))
+else:
+perm = os.stat(hostname).st_mode & 0o777
+cpio_send(cpio_header(name, perm | stat.S_IFREG,  
os.stat(hostname).st_size))

+with open(hostname, 'rb') as f:
+cpio_send(f.read())
+except:
+# We had an error uploading one of the files (e.g., one of the  
files
+# in the manifest does not exist). Let's print an error message,  
but

+# first stop the consumeoutput thread displaying the guest's output
+# so its messages do not get mixed with ours. Then, do NOT exit -
+# continue the code below to stop the guest normally.
+import time
+time.sleep(1) # give consumeoutput thread a chance to print some  
more

+stop_output = True
+print("\nERROR: file upload failed:")
+print(sys.exc_info()[1])
 cpio_send(cpio_header("TRAILER!!!", 0, 0))
 s.shutdown(socket.SHUT_WR)

 # Wait for the guest to actually finish writing and syncing
 s.recv(1)
 s.close()
+if stop_output:
+# stop_output is set when we failed during the upload, so let's
+# ha

Re: AWS new KVM hypervisor

[Avi Kivity]

On 11/16/2017 11:21 AM, Gleb Natapov wrote:

On Thu, Nov 16, 2017 at 11:15:18AM +0200, Avi Kivity wrote:

On 11/16/2017 11:03 AM, Gleb Natapov wrote:

On Thu, Nov 16, 2017 at 10:59:07AM +0200, Avi Kivity wrote:

On 11/16/2017 10:55 AM, Gleb Natapov wrote:

On Thu, Nov 16, 2017 at 10:53:52AM +0200, Avi Kivity wrote:

On 11/16/2017 10:48 AM, Nadav Har'El wrote:

On Fri, Nov 10, 2017 at 3:24 PM, Avi Kivity mailto:a...@scylladb.com>> wrote:

   OSv will need NVMe and ena drivers.


Why is that, have you seen documentation (or saw in your own tests) that
they will not support virtio-blk and virtio-net on their KVM?

It will not be supported, native drivers only.


Do you know why is that?

I can guess that they prefer to support one set of devices/drivers rather
than two. Native will have far better performance, so why bother?


New instances will have new HW, so in the end they will have to support
more than one set of drivers anyway.


What about migration?


Migration from what? Old AMIs aren't guaranteed to support virtio.


Live migration for high availability. Although I am not sure that was
supported with XEN.

Xen supports migration, But AWS does not.


Sure. I meant XEN AWS instances migration support status.


I think they could, since those "native" interfaces are actually terminated
in their Annapurna chips, not the actual hardware, so they can program those
chips to perform live migration.


Don't you need DMA "page fault" support for that.


I forgot about that.


  Do you know what is
its status?


No.


  Although with custom HW something non standard can be hacked
probably. But NVMe is not produced by Amazon.



I guess it could be hacked, but even more, that Amazon doesn't care. 
With stateless micro-services migration is needed less and less (and the 
stateful stores handle migration themselves, like ScyllaDB).


--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: AWS new KVM hypervisor

[Gleb Natapov]

On Thu, Nov 16, 2017 at 11:15:18AM +0200, Avi Kivity wrote:
> On 11/16/2017 11:03 AM, Gleb Natapov wrote:
> > On Thu, Nov 16, 2017 at 10:59:07AM +0200, Avi Kivity wrote:
> > > On 11/16/2017 10:55 AM, Gleb Natapov wrote:
> > > > On Thu, Nov 16, 2017 at 10:53:52AM +0200, Avi Kivity wrote:
> > > > > On 11/16/2017 10:48 AM, Nadav Har'El wrote:
> > > > > > On Fri, Nov 10, 2017 at 3:24 PM, Avi Kivity  > > > > > > wrote:
> > > > > > 
> > > > > >   OSv will need NVMe and ena drivers.
> > > > > > 
> > > > > > 
> > > > > > Why is that, have you seen documentation (or saw in your own tests) 
> > > > > > that
> > > > > > they will not support virtio-blk and virtio-net on their KVM?
> > > > > It will not be supported, native drivers only.
> > > > > 
> > > > Do you know why is that?
> > > I can guess that they prefer to support one set of devices/drivers rather
> > > than two. Native will have far better performance, so why bother?
> > > 
> > New instances will have new HW, so in the end they will have to support
> > more than one set of drivers anyway.
> > 
> > > >What about migration?
> > > > 
> > > Migration from what? Old AMIs aren't guaranteed to support virtio.
> > > 
> > Live migration for high availability. Although I am not sure that was
> > supported with XEN.
> 
> Xen supports migration, But AWS does not.
> 
Sure. I meant XEN AWS instances migration support status.

> I think they could, since those "native" interfaces are actually terminated
> in their Annapurna chips, not the actual hardware, so they can program those
> chips to perform live migration.
> 
Don't you need DMA "page fault" support for that. Do you know what is
its status? Although with custom HW something non standard can be hacked
probably. But NVMe is not produced by Amazon.

--
Gleb.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: AWS new KVM hypervisor

[Avi Kivity]

On 11/16/2017 11:03 AM, Gleb Natapov wrote:

On Thu, Nov 16, 2017 at 10:59:07AM +0200, Avi Kivity wrote:

On 11/16/2017 10:55 AM, Gleb Natapov wrote:

On Thu, Nov 16, 2017 at 10:53:52AM +0200, Avi Kivity wrote:

On 11/16/2017 10:48 AM, Nadav Har'El wrote:

On Fri, Nov 10, 2017 at 3:24 PM, Avi Kivity mailto:a...@scylladb.com>> wrote:

  OSv will need NVMe and ena drivers.


Why is that, have you seen documentation (or saw in your own tests) that
they will not support virtio-blk and virtio-net on their KVM?

It will not be supported, native drivers only.


Do you know why is that?

I can guess that they prefer to support one set of devices/drivers rather
than two. Native will have far better performance, so why bother?


New instances will have new HW, so in the end they will have to support
more than one set of drivers anyway.


   What about migration?


Migration from what? Old AMIs aren't guaranteed to support virtio.


Live migration for high availability. Although I am not sure that was
supported with XEN.


Xen supports migration, But AWS does not.

I think they could, since those "native" interfaces are actually 
terminated in their Annapurna chips, not the actual hardware, so they 
can program those chips to perform live migration.



--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: AWS new KVM hypervisor

[Gleb Natapov]

On Thu, Nov 16, 2017 at 10:59:07AM +0200, Avi Kivity wrote:
> On 11/16/2017 10:55 AM, Gleb Natapov wrote:
> > On Thu, Nov 16, 2017 at 10:53:52AM +0200, Avi Kivity wrote:
> > > On 11/16/2017 10:48 AM, Nadav Har'El wrote:
> > > > On Fri, Nov 10, 2017 at 3:24 PM, Avi Kivity  > > > > wrote:
> > > > 
> > > >  OSv will need NVMe and ena drivers.
> > > > 
> > > > 
> > > > Why is that, have you seen documentation (or saw in your own tests) that
> > > > they will not support virtio-blk and virtio-net on their KVM?
> > > It will not be supported, native drivers only.
> > > 
> > Do you know why is that?
> 
> I can guess that they prefer to support one set of devices/drivers rather
> than two. Native will have far better performance, so why bother?
> 
New instances will have new HW, so in the end they will have to support
more than one set of drivers anyway.

> >   What about migration?
> > 
> 
> Migration from what? Old AMIs aren't guaranteed to support virtio.
> 
Live migration for high availability. Although I am not sure that was
supported with XEN. I thought "terminate protection" means that vm will
be migrated instead of killed in case of maintenance, but now I see this
is not it.

--
Gleb.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: AWS new KVM hypervisor

[Avi Kivity]

On 11/16/2017 10:55 AM, Gleb Natapov wrote:

On Thu, Nov 16, 2017 at 10:53:52AM +0200, Avi Kivity wrote:

On 11/16/2017 10:48 AM, Nadav Har'El wrote:

On Fri, Nov 10, 2017 at 3:24 PM, Avi Kivity mailto:a...@scylladb.com>> wrote:

 OSv will need NVMe and ena drivers.


Why is that, have you seen documentation (or saw in your own tests) that
they will not support virtio-blk and virtio-net on their KVM?

It will not be supported, native drivers only.


Do you know why is that?


I can guess that they prefer to support one set of devices/drivers 
rather than two. Native will have far better performance, so why bother?



  What about migration?



Migration from what? Old AMIs aren't guaranteed to support virtio.


--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: AWS new KVM hypervisor

[Gleb Natapov]

On Thu, Nov 16, 2017 at 10:53:52AM +0200, Avi Kivity wrote:
> On 11/16/2017 10:48 AM, Nadav Har'El wrote:
> > 
> > On Fri, Nov 10, 2017 at 3:24 PM, Avi Kivity  > > wrote:
> > 
> > OSv will need NVMe and ena drivers.
> > 
> > 
> > Why is that, have you seen documentation (or saw in your own tests) that
> > they will not support virtio-blk and virtio-net on their KVM?
> 
> It will not be supported, native drivers only.
> 
Do you know why is that? What about migration?

--
Gleb.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: AWS new KVM hypervisor

[Avi Kivity]

On 11/16/2017 10:48 AM, Nadav Har'El wrote:


On Fri, Nov 10, 2017 at 3:24 PM, Avi Kivity > wrote:


OSv will need NVMe and ena drivers.


Why is that, have you seen documentation (or saw in your own tests) 
that they will not support virtio-blk and virtio-net on their KVM?


It will not be supported, native drivers only.

--
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: AWS new KVM hypervisor

[Nadav Har'El]

On Fri, Nov 10, 2017 at 3:24 PM, Avi Kivity  wrote:

> OSv will need NVMe and ena drivers.
>

Why is that, have you seen documentation (or saw in your own tests) that
they will not support virtio-blk and virtio-net on their KVM?


>
> On Nov 10, 2017 15:07, "Waldek Kozaczuk"  wrote:
>
> https://www.theregister.co.uk/2017/11/07/aws_writes_new_kvm_
> based_hypervisor_to_make_its_cloud_go_faster/ - I wonder what impact it
> makes to OSv?
>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.